Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 17, 2021.

A friend recently shared a sharing of a screenshot of a reblogging of a reblogging of this tumblr post:

Pro-Capitalist's defense of capitalism is just explaining how it works, and then when you say "yes I know, I just think it shouldn't be like that" they explain it to you again but angrier this time. —strawberry-crocodile.

I really like this perspective, even as someone relatively pro-capitalism, because I think it captures something that often goes wrong in these discussions.

The strongest argument in favor of capitalism is that in practice it works for most things, better than the other systems we've tried. Not because it was designed to work, but because that's just how it falls together. When someone points at a piece of the system that seems unfair or wasteful and says "I just think it shouldn't be like that," stopping it's going to have effects elsewhere in the system, often negative ones. And so pro-capitalism folks often respond by trying to explain capitalism harder: what role is the thing you want to change filling? When people propose removing something without engaging with how it ties in to the rest of the system, it is natural to assume they don't know about its function and try to explain.

As in the opening quote, however, people don't want more explanation of the workings of the status quo. Instead, I think a better response is to think about what you expect would go wrong, and ask if they would expect that. Perhaps they don't, and you can try and figure out where specifically your expectations diverge. Perhaps they do, and they think it's worth it. Perhaps they have additional proposals which work together. Whichever way the conversation goes, I think it probably is more productive?

(Overall my perspective is that while things are much worse than they could be, they're also much better than they have ever been. I really don't want us to break the system that keeps improving our ability to turn time and stuff into what people need. At the same time, to the extent that we can do it without breaking this cycle of improvement, I'd like to see far more redistribution of wealth. In my own life this looks like giving.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Low-Hanging (Monetary) Fruit for Wealthy EAs, published by MichaelDickens on October 16, 2021 on The Effective Altruism Forum.

Cross-posted to my website.

Ordinary wealthy people don't care as much about getting more money because they already have a lot of it. So we should expect to be able to find overlooked methods for rich people to get richer.[1] Wealthy effective altruists might value their billionth dollar nearly as much as their first dollar, so they should seek out these overlooked methods.

If someone got rich doing X (where X = starting a startup, excelling at a high-paying profession, etc.), their best way of making money on the margin might not be to do more X. It might be to do something entirely different.

Some examples:

Sam Bankman-Fried increased his net worth by $10,000,000,000 in four years by founding FTX. He earned most of those zeroes by doing the hard work of starting a company, and there's no shortcut around that. But, importantly, he managed to retain most of his original stake in FTX. For most founders, by the time their company is worth $10 billion or more, they only own maybe 10% of it. If Sam had given away a normal amount of equity to VCs, he might have only gotten $2 billion from FTX instead of $10 billion. In some sense, 80% of the money he earned from FTX came purely from retaining equity.[2]

(Bill Gates might be another person who got richer than usual by retaining ownership.[3])

Warren Buffett is an investing genius. But if he had only been an investing genius, he would have made a lot less money. In addition to earning high investment returns, he borrowed money at below-market interest rates using insurance float. That let him compound his returns much faster than he otherwise would have.

Long-Term Capital Management was an extremely successful hedge fund (at least until it blew up, but let's not talk about that part). Much of its success didn't come from its trading strategies, but from the fact that it negotiated unusually favorable deals with its banking partners.

A quasi-example: Instead of trying to make more money, try to (legally) pay less tax. This idea is not exactly overlooked, so it's not a central example of what I'm talking about. But I have noticed that most "normal-level" rich people (with between $1 million and $10 million) don't pay nearly enough attention to taxes. I know a lot of people who could save $10,000 to $100,000 per year with 20 hours of work or less.

In these examples, it's not necessarily that the person (or company) came up with a trick that nobody else had thought of. Obviously every startup founder would love to retain as much of their stock as they can. But they usually have to choose between giving up equity and running out of funding, and they rightly choose to keep their company alive. My point isn't that it's easy. My point is that, on the margin, EA founders should probably pay more attention to retaining equity. Maybe a founder could do some extra work and find a way to keep 11% of their equity instead of 10%, without meaningfully reducing their startup's chances of success. Getting that extra 1 percentage point of equity might be a lot easier than raising the company's ultimate valuation by 10%.

Wealthy EAs care more about marginal dollars than ordinary wealthy people, and they can take advantage of that fact. They can look for (relatively) easy ways of getting money. I only spent about five minutes coming up with the examples I gave, so I'm sure there are plenty of other ideas.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Low-Hanging (Monetary) Fruit for Wealthy EAs, published by MichaelDickens on October 16, 2021 on The Effective Altruism Forum.

Cross-posted to my website.

Ordinary wealthy people don't care as much about getting more money because they already have a lot of it. So we should expect to be able to find overlooked methods for rich people to get richer.[1] Wealthy effective altruists might value their billionth dollar nearly as much as their first dollar, so they should seek out these overlooked methods.

If someone got rich doing X (where X = starting a startup, excelling at a high-paying profession, etc.), their best way of making money on the margin might not be to do more X. It might be to do something entirely different.

Some examples:

Sam Bankman-Fried increased his net worth by $10,000,000,000 in four years by founding FTX. He earned most of those zeroes by doing the hard work of starting a company, and there's no shortcut around that. But, importantly, he managed to retain most of his original stake in FTX. For most founders, by the time their company is worth $10 billion or more, they only own maybe 10% of it. If Sam had given away a normal amount of equity to VCs, he might have only gotten $2 billion from FTX instead of $10 billion. In some sense, 80% of the money he earned from FTX came purely from retaining equity.[2]

(Bill Gates might be another person who got richer than usual by retaining ownership.[3])

Warren Buffett is an investing genius. But if he had only been an investing genius, he would have made a lot less money. In addition to earning high investment returns, he borrowed money at below-market interest rates using insurance float. That let him compound his returns much faster than he otherwise would have.

Long-Term Capital Management was an extremely successful hedge fund (at least until it blew up, but let's not talk about that part). Much of its success didn't come from its trading strategies, but from the fact that it negotiated unusually favorable deals with its banking partners.

A quasi-example: Instead of trying to make more money, try to (legally) pay less tax. This idea is not exactly overlooked, so it's not a central example of what I'm talking about. But I have noticed that most "normal-level" rich people (with between $1 million and $10 million) don't pay nearly enough attention to taxes. I know a lot of people who could save $10,000 to $100,000 per year with 20 hours of work or less.

In these examples, it's not necessarily that the person (or company) came up with a trick that nobody else had thought of. Obviously every startup founder would love to retain as much of their stock as they can. But they usually have to choose between giving up equity and running out of funding, and they rightly choose to keep their company alive. My point isn't that it's easy. My point is that, on the margin, EA founders should probably pay more attention to retaining equity. Maybe a founder could do some extra work and find a way to keep 11% of their equity instead of 10%, without meaningfully reducing their startup's chances of success. Getting that extra 1 percentage point of equity might be a lot easier than raising the company's ultimate valuation by 10%.

Wealthy EAs care more about marginal dollars than ordinary wealthy people, and they can take advantage of that fact. They can look for (relatively) easy ways of getting money. I only spent about five minutes coming up with the examples I gave, so I'm sure there are plenty of other ideas.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 16, 2021.

I appreciate Zoe Curzi's revelations of her experience with Leverage. I know how hard it is to speak up when no or few others do, and when people are trying to keep things under wraps.

I haven't posted much publicly about my experiences working as a researcher at MIRI (2015-2017) or around CFAR events, to a large degree because I've been afraid. Now that Zoe has posted about her experience, I find it easier to do so, especially after the post was generally well-received by LessWrong.

I felt moved to write this, not just because of Zoe's post, but also because of Aella's commentary:

I've found established rationalist communities to have excellent norms that prevent stuff like what happened at Leverage. The times where it gets weird is typically when you mix in a strong leader + splintered, isolated subgroup + new norms. (this is not the first time)

This seemed to me to be definitely false, upon reading it. Most of what was considered bad about the events at Leverage Research also happened around MIRI/CFAR, around the same time period (2017-2019).

I don't want to concentrate on the question of which is "worse"; it is hard to even start thinking about that without discussing facts on the ground and general social models that would apply to both cases. I also caution against blame in general, in situations like these, where many people (including me!) contributed to the problem, and have kept quiet for various reasons. With good reason, it is standard for truth and reconciliation events to focus on restorative rather than retributive justice, and include the possibility of forgiveness for past crimes.

As a roadmap for the rest of the post, I'll start by describing some background, describe some trauma symptoms and mental health issues I and others have experienced, and describe the actual situations that these mental events were influenced by and "about" to a significant extent.

Background: choosing a career

After I finished my CS/AI Master's degree at Stanford, I faced a choice of what to do next. I had a job offer at Google for machine learning research and a job offer at MIRI for AI alignment research. I had also previously considered pursuing a PhD at Stanford or Berkeley; I'd already done undergrad research at CoCoLab, so this could have easily been a natural transition.

I'd decided against a PhD on the basis that research in industry was a better opportunity to work on important problems that impact the world; since then I've gotten more information from insiders that academia is a "trash fire" (not my quote!), so I don't regret this decision.

I was faced with a decision between Google and MIRI. I knew that at MIRI I'd be taking a pay cut. On the other hand, I'd be working on AI alignment, an important problem for the future of the world, probably significantly more important than whatever I'd be working on at Google. And I'd get an opportunity to work with smart, ambitious people, who were structuring their communication protocols and life decisions around the content of the LessWrong Sequences.

These Sequences contained many ideas that I had developed or discovered independently, such as functionalist theory of mind, the idea that Solomonoff Induction was a formalization of inductive epistemology, and the idea that one-boxing in Newcomb's problem is more rational than two-boxing. The scene attracted thoughtful people who cared about getting the right answer on abstract problems like this, making for very interesting conversations.

Research at MIRI was an extension of such interesting conversations to rigorous mathematical formalism, making it very fun (at least for a time). Some of the best research I've done was at MIRI (reflective oracles, logical induction, others). I met many of my current friends through LessWrong, MIRI, and the broader...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 16, 2021.

I appreciate Zoe Curzi's revelations of her experience with Leverage. I know how hard it is to speak up when no or few others do, and when people are trying to keep things under wraps.

I haven't posted much publicly about my experiences working as a researcher at MIRI (2015-2017) or around CFAR events, to a large degree because I've been afraid. Now that Zoe has posted about her experience, I find it easier to do so, especially after the post was generally well-received by LessWrong.

I felt moved to write this, not just because of Zoe's post, but also because of Aella's commentary:

I've found established rationalist communities to have excellent norms that prevent stuff like what happened at Leverage. The times where it gets weird is typically when you mix in a strong leader + splintered, isolated subgroup + new norms. (this is not the first time)

This seemed to me to be definitely false, upon reading it. Most of what was considered bad about the events at Leverage Research also happened around MIRI/CFAR, around the same time period (2017-2019).

I don't want to concentrate on the question of which is "worse"; it is hard to even start thinking about that without discussing facts on the ground and general social models that would apply to both cases. I also caution against blame in general, in situations like these, where many people (including me!) contributed to the problem, and have kept quiet for various reasons. With good reason, it is standard for truth and reconciliation events to focus on restorative rather than retributive justice, and include the possibility of forgiveness for past crimes.

As a roadmap for the rest of the post, I'll start by describing some background, describe some trauma symptoms and mental health issues I and others have experienced, and describe the actual situations that these mental events were influenced by and "about" to a significant extent.

Background: choosing a career

After I finished my CS/AI Master's degree at Stanford, I faced a choice of what to do next. I had a job offer at Google for machine learning research and a job offer at MIRI for AI alignment research. I had also previously considered pursuing a PhD at Stanford or Berkeley; I'd already done undergrad research at CoCoLab, so this could have easily been a natural transition.

I'd decided against a PhD on the basis that research in industry was a better opportunity to work on important problems that impact the world; since then I've gotten more information from insiders that academia is a "trash fire" (not my quote!), so I don't regret this decision.

I was faced with a decision between Google and MIRI. I knew that at MIRI I'd be taking a pay cut. On the other hand, I'd be working on AI alignment, an important problem for the future of the world, probably significantly more important than whatever I'd be working on at Google. And I'd get an opportunity to work with smart, ambitious people, who were structuring their communication protocols and life decisions around the content of the LessWrong Sequences.

These Sequences contained many ideas that I had developed or discovered independently, such as functionalist theory of mind, the idea that Solomonoff Induction was a formalization of inductive epistemology, and the idea that one-boxing in Newcomb's problem is more rational than two-boxing. The scene attracted thoughtful people who cared about getting the right answer on abstract problems like this, making for very interesting conversations.

Research at MIRI was an extension of such interesting conversations to rigorous mathematical formalism, making it very fun (at least for a time). Some of the best research I've done was at MIRI (reflective oracles, logical induction, others). I met many of my current friends through LessWrong, MIRI, and the broader...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Optimization Concepts in the Game of Life, published by Vika on October 16, 2021 on The AI Alignment Forum.

Abstract: We define robustness and retargetability (two of Flint’s measures of optimization) in Conway’s Game of Life and apply the definitions to a few examples. The same approach likely works in most embedded settings, and provides a frame for conceptualizing and quantifying these aspects of agency. We speculate on the relationship between robustness and retargetability, and identify various directions for future work.

Motivation

We would like to better understand the fundamental principles of agency (and related phenomena including optimization and goal-directedness). We focus on agency because we believe agency is a core source of risk from AI systems, especially in worlds with one (or few) most-capable systems. The goals of the most competent consequence-driven systems are more likely to be achieved, because trying outperforms not trying or less competent trying. We do not want to create a world where such systems are working against us. By better understanding agency, we hope to improve our ability to avoid mistakenly building systems working capably against us, and to correct course if we do.

A rich source of confusions about agency comes from attending to the fact that goal-directed systems are part of – embedded in – the environment that their goals are about. Most practical work on AI avoids the confusions of embedded agency by constructing and enforcing a Cartesian boundary between agent and environment, using frameworks such as reinforcement learning (RL) that define an interaction protocol. We focus on embedded agency because we expect not to be able to enforce a Cartesian boundary for highly capable agents in general domains, and, as a particularly strong instance of this, because agents may emerge unexpectedly in systems where we did not design how they interface with the rest of the world.

Our approach to deconfusion in this post is to identify concepts that seem relevant to embedded agency but do not have technical definitions, to propose some definitions, and see how they fare on some examples. More generally, we are interested in analyzing small examples of agency-related phenomena in the hope that some examples will be simple enough to yield insight while retaining essential features of the phenomenon.

Optimization in the Game of Life

Concepts

We draw two concepts from Alex Flint’s essay The Ground of Optimization. Flint defines an optimizing system as a system that evolves towards a small set of target configurations from a broad basin of attraction, despite perturbations. The essay introduces measures for quantifying optimization systems. One is robustness: how robust to perturbations is the process of reaching the target set, e.g. the number of dimensions on which perturbations can be made or the magnitude of the perturbations. Another measure is retargetability: whether the system can be transformed into another optimizing system with a different target configuration set via a small change.

Here, we develop more precise definitions of these concepts by concentrating on a particular concrete domain: Conway’s Game of Life. This is a natural setting for studying embedded agency because it is a deterministic environment with no pre-specified Cartesian boundaries, which is rich enough to support emergent goal-directed behavior, yet simple enough to define the concepts above explicitly.

Examples

Before getting to the definitions, let’s look at how we might draw analogies between some of the examples of systems (including optimizing systems) from the Ground of Optimization post and structures in the Game of Life.

4-square still life

Glider

Eater

A 4-square still life is like a bottle cap in that it has been designed (or selected) to stay in plac...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Optimization Concepts in the Game of Life, published by Vika on October 16, 2021 on The AI Alignment Forum.

Abstract: We define robustness and retargetability (two of Flint’s measures of optimization) in Conway’s Game of Life and apply the definitions to a few examples. The same approach likely works in most embedded settings, and provides a frame for conceptualizing and quantifying these aspects of agency. We speculate on the relationship between robustness and retargetability, and identify various directions for future work.

Motivation

We would like to better understand the fundamental principles of agency (and related phenomena including optimization and goal-directedness). We focus on agency because we believe agency is a core source of risk from AI systems, especially in worlds with one (or few) most-capable systems. The goals of the most competent consequence-driven systems are more likely to be achieved, because trying outperforms not trying or less competent trying. We do not want to create a world where such systems are working against us. By better understanding agency, we hope to improve our ability to avoid mistakenly building systems working capably against us, and to correct course if we do.

A rich source of confusions about agency comes from attending to the fact that goal-directed systems are part of – embedded in – the environment that their goals are about. Most practical work on AI avoids the confusions of embedded agency by constructing and enforcing a Cartesian boundary between agent and environment, using frameworks such as reinforcement learning (RL) that define an interaction protocol. We focus on embedded agency because we expect not to be able to enforce a Cartesian boundary for highly capable agents in general domains, and, as a particularly strong instance of this, because agents may emerge unexpectedly in systems where we did not design how they interface with the rest of the world.

Our approach to deconfusion in this post is to identify concepts that seem relevant to embedded agency but do not have technical definitions, to propose some definitions, and see how they fare on some examples. More generally, we are interested in analyzing small examples of agency-related phenomena in the hope that some examples will be simple enough to yield insight while retaining essential features of the phenomenon.

Optimization in the Game of Life

Concepts

We draw two concepts from Alex Flint’s essay The Ground of Optimization. Flint defines an optimizing system as a system that evolves towards a small set of target configurations from a broad basin of attraction, despite perturbations. The essay introduces measures for quantifying optimization systems. One is robustness: how robust to perturbations is the process of reaching the target set, e.g. the number of dimensions on which perturbations can be made or the magnitude of the perturbations. Another measure is retargetability: whether the system can be transformed into another optimizing system with a different target configuration set via a small change.

Here, we develop more precise definitions of these concepts by concentrating on a particular concrete domain: Conway’s Game of Life. This is a natural setting for studying embedded agency because it is a deterministic environment with no pre-specified Cartesian boundaries, which is rich enough to support emergent goal-directed behavior, yet simple enough to define the concepts above explicitly.

Examples

Before getting to the definitions, let’s look at how we might draw analogies between some of the examples of systems (including optimizing systems) from the Ground of Optimization post and structures in the Game of Life.

4-square still life

Glider

Eater

A 4-square still life is like a bottle cap in that it has been designed (or selected) to stay in plac...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 15, 2021.

Suppose you want to argue for the longtermist importance of shaping AI development.

My key point is: you can choose to make an argument that is more specific, or more general. Specific arguments include more details about what AI development looks like and how exactly it threatens humanity’s longterm potential; general arguments include fewer. The most specific arguments specify a complete threat model.

I think this is an important distinction, and that people being more clear about what kind of argument they are making/motivated by would improve the quality of discussion about how much AI should be prioritised as a cause area, and how to prioritise between different AI-related interventions.

The distinction

I'll clarify the distinction by giving examples of arguments that sit near the ends of the general vs specific spectrum.

Here's a highly specific argument:

If TAI arrives this century, then a scenario broadly similar to What failure looks like is plausible (see here for a summary if you’re unfamiliar)

In this scenario, most of the value of the longterm future is lost

Here's a highly general argument:

It's plausible that TAI will arrive this century

If TAI arrives this century, it's probably one of our best bets for positively shaping the longterm future

Of course, many arguments fall somewhere in between these extremes. For example, Richard Ngo's AGI safety from first principles argues specifically for the plausibility of "AI takeover" (a scenario where the most consequential decisions about the future get made by AI systems with goals that aren’t desirable by human standards) - which is just one among many possible risks from advanced AI. And many of its arguments won't apply if the field of AI moves away from focusing on machine learning. But the argument isn't fully specific, because the author doesn't argue for the plausibility of any specific variety of AI takeover scenario (there are many), instead focusing on making a more general case for the plausibility of AI takeover.

Joe Carlsmith's report on existential risk from power-seeking AI is similar, and also falls somewhere between the extremes. As do arguments that appeal to the inner alignment problem being necessary for AI existential safety, but very difficult.

Why this distinction matters

Mostly, for the reason I mentioned at the beginning: when arguing about the longtermist importance of shaping AI development, I think that if people were clearer about how general/specific their arguments are/need to be, then this would improve discussion about (1) how much AI should be prioritised as a cause area, and (2) how to prioritise between different AI-related interventions.

For an example of (1): I sometimes hear people arguing against longtermist case for shaping AI development by pointing out that there’s wide disagreement and uncertainty about which risk scenarios are actually plausible, as if this is an argument against the entire case - when actually it’s just an argument against one kind of case you can make. Conversely, I’ve heard other discussions being vaguer than they should be about how exactly AI leads to existential catastrophe.

For an example of (2): for prioritising interventions within AI as a cause area, if the strongest arguments for this kind of work are very general, this suggests a broader portfolio of work than if the strongest arguments pick out a particular class of threat models.

Finally, I personally think that the strongest case that we can currently make for the longtermist importance of shaping AI development is fairly general - something along the lines of the most important century series - and yet this doesn't seem to be the "default" argument (i.e. the one presented in key EA content/fellowships/etc. when discussing AI). Instead, it seems the "default" is...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 15, 2021.

Suppose you want to argue for the longtermist importance of shaping AI development.

My key point is: you can choose to make an argument that is more specific, or more general. Specific arguments include more details about what AI development looks like and how exactly it threatens humanity’s longterm potential; general arguments include fewer. The most specific arguments specify a complete threat model.

I think this is an important distinction, and that people being more clear about what kind of argument they are making/motivated by would improve the quality of discussion about how much AI should be prioritised as a cause area, and how to prioritise between different AI-related interventions.

The distinction

I'll clarify the distinction by giving examples of arguments that sit near the ends of the general vs specific spectrum.

Here's a highly specific argument:

If TAI arrives this century, then a scenario broadly similar to What failure looks like is plausible (see here for a summary if you’re unfamiliar)

In this scenario, most of the value of the longterm future is lost

Here's a highly general argument:

It's plausible that TAI will arrive this century

If TAI arrives this century, it's probably one of our best bets for positively shaping the longterm future

Of course, many arguments fall somewhere in between these extremes. For example, Richard Ngo's AGI safety from first principles argues specifically for the plausibility of "AI takeover" (a scenario where the most consequential decisions about the future get made by AI systems with goals that aren’t desirable by human standards) - which is just one among many possible risks from advanced AI. And many of its arguments won't apply if the field of AI moves away from focusing on machine learning. But the argument isn't fully specific, because the author doesn't argue for the plausibility of any specific variety of AI takeover scenario (there are many), instead focusing on making a more general case for the plausibility of AI takeover.

Joe Carlsmith's report on existential risk from power-seeking AI is similar, and also falls somewhere between the extremes. As do arguments that appeal to the inner alignment problem being necessary for AI existential safety, but very difficult.

Why this distinction matters

Mostly, for the reason I mentioned at the beginning: when arguing about the longtermist importance of shaping AI development, I think that if people were clearer about how general/specific their arguments are/need to be, then this would improve discussion about (1) how much AI should be prioritised as a cause area, and (2) how to prioritise between different AI-related interventions.

For an example of (1): I sometimes hear people arguing against longtermist case for shaping AI development by pointing out that there’s wide disagreement and uncertainty about which risk scenarios are actually plausible, as if this is an argument against the entire case - when actually it’s just an argument against one kind of case you can make. Conversely, I’ve heard other discussions being vaguer than they should be about how exactly AI leads to existential catastrophe.

For an example of (2): for prioritising interventions within AI as a cause area, if the strongest arguments for this kind of work are very general, this suggests a broader portfolio of work than if the strongest arguments pick out a particular class of threat models.

Finally, I personally think that the strongest case that we can currently make for the longtermist importance of shaping AI development is fairly general - something along the lines of the most important century series - and yet this doesn't seem to be the "default" argument (i.e. the one presented in key EA content/fellowships/etc. when discussing AI). Instead, it seems the "default" is...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 13, 2021.

This post was written by Devin Kalish, my co-blogger who does not have an account on the EA Forum. I'm posting this on his behalf, and will reply to comments with his replies if/when he responds to them.

Epistemic Status: Uh, I haven't written one of these before, but since I'm cross-posting to EA Forum, I will disclaim in advance that I banged the first draft of this out in like an hour when I should have been studying, and the empirical claims about movements in it I give examples of aren't supported more deeply. I rely on the readers to share similar impressions to me on reflection, so don't treat this as any great confirmation of those impressions.

Special thanks to Glen Weyl for offering some comments on this piece during this very busy time for him.

Summary (added Oct 13, 2021): I wrote a good deal of this stream of conscious so it isn’t really structured as an argument. More a way of me to connect some personal thoughts/experiences together in a hopefully productive way. I can see how that wouldn’t be super accessible. The basic argument embedded in it though is:

Effective Altruism, like many idealistic movements, started out taking critics very very seriously and trying to reach out to/be charitable to them as much as possible, which is a good thing.

Effective Altruism, like most movements that grow older, is not quite like that anymore, it seems to respond with less frequency and generosity to critics than it used to, which is unfortunate but understandable.

Understandable as it is, we should at least take a bit more notice of it if that’s the path we are going down because.

Many movements move on from here to ridiculing criticisms by treating common criticisms as though they were obviously, memeably false, and that everyone in the know gets that (I didn’t use examples, but the one most on my mind was the midwit meme format, which only requires the argument being ridiculed, your stated, undefended position, and some cartoons, to make it look like you’ve made a point). This is bad and we should be careful not to start doing it.

The post proper:

The Effective Altruism movement is not that old, but it is not that new either. There are various ways it has changed, but I wanted to share a little bit of nostalgia. Back in the youth of the movement in 2013, the CEO and a consultant from Charity Navigator published a hit piece on Effective Altruism that is now somewhat notorious in the movement, in no small part, for being pretty bad[1]. It makes a few highly abstracted, melodramatic arguments that could be countered with simple case comparisons, it engages in needless name-calling (defective altruism), and to boot it was written by people with an extremely obvious professional conflict of interest with the movement. And yet, William MacAskill read this, emailed its authors to isolate their core intellectual disagreement, and then published a brief, pretty polite response to it in the same publication.

This was not the only time something like this happened, a few months later he had an exchange with Warren Buffet's son, which was also polite and seems to me more charitable than the quality of the arguments really warranted (though it was on the heels of a much less polite exchange).

Fast-forward to 2021. Phil Torres writes a piece for Current Affairs tearing into longtermism, one of MacAskill's biggest interests and a topic he's personally contributed to within Effective Altruist thought. In my opinion, it also isn't great, but it is better than these earlier critiques on any dimension you could care to measure. It makes intellectually interesting points about the philosophy[2], it gets specific, and it quotes major figures and works within the field. Far from coming from an outsider who has barely thought about this and who has a conflict of ...