Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental Convergence For Realistic Agent Objectives, published by Alex Turner on January 22, 2022 on The AI Alignment Forum.

The current power-seeking theorems say something like:

Give me a utility function, any utility function, and for most ways I could jumble it up—most ways I could permute which outcomes get which utility, for most of these permutations, the agent will seek power.

This kind of argument assumes that (the set of utility functions we might specify) is closed under permutation. This is unrealistic, because practically speaking we reward agents based off of observed features of the agent's environment.

For example, Pac-Man eats dots and gains points. A football AI scores a touchdown and gains points. A robot hand solves a Rubik's cube and gains points. But most permutations of these objectives are implausible because they're high-entropy, they're very complex, they assign high reward to one state and low reward to another state without a simple generating rule that grounds out in observed features. Practical objective specification doesn't allow that many degrees of freedom in what states get what reward.

I explore how instrumental convergence works in this case. I also walk through how these new results retrodict the fact that instrumental convergence basically disappears for agents with utility functions over action-observation histories.

Case Studies

Gridworld

Consider the following environment, where the agent can either stay put or move along a purple arrow.

Suppose the agent gets some amount of reward each timestep, and it's choosing a policy to maximize its average per-timestep reward. Previous results tell us that for generic reward functions over states, at least half of them incentivize going right. There are two terminal states on the left, and three on the right, and 3 > 2; we conclude that at least floor(3/2)floor(3/2)+1=12 of objectives incentivize going right.

But it's damn hard to have so many degrees of freedom that you're specifying a potentially independent utility number for each state. Meaningful utility functions will be featurized in some sense—only depending on certain features of the world state, and of how the outcomes transpired, etc. If the featurization is linear, then it's particularly easy to reason about power-seeking incentives.

Let feat(s):=⎛⎜⎝1 if s = △, 0 else1 if s = ◯, 0 else1 if s = ★, 0 else⎞⎟⎠. That is, the featurization only cares what shape the agent is standing on. Suppose the agent makes decisions in a way which depends only on the featurized reward of a state: R(s)=feat(s)⊤α, where α∈R3 expresses the feature coefficients. Then the relevant terminal states are only {triangle, circle, star}, and we conclude that 23 of coefficient vectors incentivize going right. This is true more precisely in the orbit sense: For every coefficient vector α, at least 23 of its permuted variants make the agent prefer to go right.

This particular featurization increases the strength of the orbit-level incentives—whereas before, we could only guarantee 12-strength power-seeking tendency, now we guarantee 23-level.

There's another point I want to make in this tiny environment.

Suppose we find an environmental symmetry ϕ which lets us apply the original power-seeking theorems to raw reward functions over the world state. Letting es∈R6 be a column vector with an entry of 1 at state s and 0 elsewhere, in this environment, we have the symmetry enforced by ϕ⋅State distributions, left{e△,eleft}={e◯,eright}⊊State distributions, right{e◯,eright,e★}.

Given a state featurization, and given that we know that there's a state-level environmental symmetry ϕ, when can we conclude that there's also feature-level power-seeking in the environment?

Here, we're asking "if reward is only allowed to depend on how often the agent visits each shape,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Structure of the Guild of the Rose, published by moridinamael on January 21, 2022 on LessWrong.

This is a link post for The Structure of the Guild, part one of a series of ongoing retrospectives on the progress of the Guild of the Rose, an organization for rationalist education and community.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [AN #171]: Disagreements between alignment "optimists" and "pessimists", published by Rohin Shah on January 21, 2022 on The AI Alignment Forum.

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

1. We are very likely going to keep improving AI capabilities until we reach AGI, at which point either the world is destroyed, or we use the AI system to take some pivotal act before some careless actor destroys the world.

2. In either case, the AI system must be producing high-impact, world-rewriting plans; such plans are “consequentialist” in that the simplest way to get them (and thus, the one we will first build) is if you are forecasting what might happen, thinking about the expected consequences, considering possible obstacles, searching for routes around the obstacles, etc. If you don’t do this sort of reasoning, your plan goes off the rails very quickly - it is highly unlikely to lead to high impact. In particular, long lists of shallow heuristics (as with current deep learning systems) are unlikely to be enough to produce high-impact plans.

3. We’re producing AI systems by selecting for systems that can do impressive stuff, which will eventually produce AI systems that can accomplish high-impact plans using a general underlying “consequentialist”-style reasoning process (because that’s the only way to keep doing more impressive stuff). However, this selection process does not constrain the goals towards which those plans are aimed. In addition, most goals seem to have convergent instrumental subgoals like survival and power-seeking that would lead to extinction. This suggests that we should expect an existential catastrophe by default.

4. None of the methods people have suggested for avoiding this outcome seem like they actually avert this story.

Richard responds to this with a few distinct points:

1. It might be possible to build AI systems which are not of world-destroying intelligence and agency, that humans use to save the world. For example, we could make AI systems that do better alignment research. Such AI systems do not seem to require the property of making long-term plans in the real world in point (3) above, and so could plausibly be safe.

2. It might be possible to build general AI systems that only state plans for achieving a goal of interest that we specify, without executing that plan.

3. It seems possible to create consequentialist systems with constraints upon their reasoning that lead to reduced risk.

4. It also seems possible to create systems with the primary aim of producing plans with certain properties (that aren't just about outcomes in the world) -- think for example of corrigibility (AN #35) or deference to a human user.

5. (Richard is also more bullish on coordinating not to use powerful and/or risky AI systems, though the debate did not discuss this much.)

Eliezer’s responses:

1. AI systems that help with alignment research to such a degree that it actually makes a difference are almost certainly already dangerous.

2. It is the plan itself that is risky; if the AI system made a plan for a goal that wasn’t the one we actually meant, and we don’t understand that plan, that plan can still cause extinction. It is the misaligned optimization that produced the plan that is dangerous.

3 and 4. It is certainly possible to do such things; the space of minds that could be designed is very large. However, it is difficult to do such things, as they tend ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Suggestion: Effective Animal Advocacy forum, published by saulius on January 21, 2022 on The Effective Altruism Forum.

Idea:

A forum that only shows animal-welfare-related EA forum posts. The relationship between this Effective Animal Advocacy (EAA) forum and the EA forum could be similar to the relationship between the AI Alignment Forum and LessWrong. I don’t fully understand what that relationship is, but it seems that if you post or comment on the AI alignment, the post appears on LessWrong too. We could kick-start the EAA forum by showing EA forum posts with certain tags (e.g. Farmed animal welfare) in this new forum. An ability to post only on the EAA forum without posting on the EA forum would also be nice. I talked about this with a bunch of EAA researchers yesterday and all seem to agree that this would be a good idea.

What could it improve:

TL; DR: Very few animal activists comment and post in the EAA forum. I think this is because they don’t feel like the EA forum is their community, and most of the content here is irrelevant to them. A forum only for animal activists might get more engagement from people who do direct work on animal issues. Without such people participating in our discussions on how to solve animal issues, we are less likely to come up with viable solutions.

Target audiences of EAA posts on EA forum include:

EAA funders,

Employees at animal charities (especially decision-makers),

People who want to work in animal advocacy,

EAA researchers,

Some people who work on animal-related policy,

Other animal activists and volunteers.

However, most of the comments and and posts on the EAA posts on the EA forum seem to be from very few highly engaged people who are part of both EA and animal advocacy movements. Most of these people in the categories above (especially employees at animal charities) are not engaged with other parts of the EA movement. In a way, they don’t have much reason to do that because they already chose the issue they want to focus on, and the other issues are just distractions.

The way I imagine it, if someone sends them an EA forum post that’s relevant, they will read it. But if they then go to the EA forum frontpage or something, they see a bunch of posts that are mostly irrelevant for them. I set up my tag filters to almost maximally filter for animal-welfare related posts and still in the frontpage I didn’t see a single animal-relevant post just now until I pressed “Read more”.

Sure, they can press on a tag, but those are mostly old posts. They might be irrelevant now and if they comment on such a post, barely anyone will notice so there isn’t much incentive to do so. Hence, they soon close the EA forum. They probably don’t regularly check the EA forum to see what is new here.

Also, I imagine it might feel scary for animal advocates who are not engaged with other parts of EA to comment and post in the EA forum because they don’t understand what kind of people will read it, how those people think, what kinds of posts and comments they want to see, and what are the social norms in the EA community. Because EA is not their community. Animal advocacy is their community. Most of these people have colleagues and friends who are animal advocates. Hence they understand what kind of knowledge and ideas are relevant and valued in these circles. Hence, they also have much better intuitions about what kinds of posts and comments would be useful to post, and would be well-received, which I think would make them more likely to post.

EAA really needs new ideas. We should try more strategies and see which of them work. My impression is that good ideas is a bottleneck, as there would be probably be enough funding and talent to pursue those ideas. If we had a forum that includes both EA researchers and people who are directly working on solvi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Get In The Van, published by ThomasWoodside on January 21, 2022 on The Effective Altruism Forum.

Thanks to Jessica McCurdy, Buck Shlegeris, Sofya Lebedeva, Matt Burtell, Julia Wise, Aris Richardson, and Eui Young Kim for feedback on this post.

People commonly ask me how I got involved with EA. My answer is somewhat unusual.

I got in a van.

It was fall 2019, relatively early in my first year of undergrad at Yale. I was subscribed to the Yale EA newsletter (as I was for dozens of extracurriculars), but I wasn’t involved in the group. One day something on the newsletter piqued my interest: “People with backgrounds from several effective altruism organizations, such as MIRI and Open Phil, are coming to Yale!” They promised to offer “career advice” among other things.

What was this “MIRI”? What was “Open Phil”? Who were “Buck Shlegeris” and “Asya Bergal”? I had no idea. But I did know that I wanted to make an impact with my career, and EA was associated with that in my head. I did know I was interested in AI, and this “MIRI” thing seemed related. And I did know I wanted a summer internship.

For whatever reason, I ended up talking to Buck, who apparently worked for MIRI. Here is how I described my conversation to a friend, memorialized in my text messages (do not take this as an accurate portrayal of Buck’s beliefs):

This was weird. Here was a guy telling me about how we were going to be turned into paperclips if we weren’t careful. That he worked for a whole organization devoted to preventing us from being turned into paperclips (this is not an accurate portrayal of MIRI’s goals).

But then, they said they had to leave. They were sorry to have been so late, but they had to go on the next part of their road trip.

“Does anyone want to come with us to New York City?”

Two students volunteered themselves pretty quickly. They were both graduate students and already somewhat involved with Yale EA. I didn’t know either of them. In fairness, I had no idea who any of these people were.

I thought for a few minutes, and I decided: I’m going to get in their van. And so I did.

At one point in the van, I asked Buck: “so, where are we going?” He laughed and seemed to think it was very funny that I had gotten in a van without even knowing where it was going.

“A Slate Star Codex meetup.”

I was puzzled at the strange name. “What’s that?”

We arrived as the sun was setting. The meetup was in a park, and there were many people milling around. Apparently, “Scott Alexander himself” was at the meetup. I think somebody pointed him out to me. It’s not like I remember though, because I had no idea who Scott Alexander was.

I could describe in more detail the conversations I had at the meetup, but that isn't the point of this piece. It is important to note, though, that I didn't think that everyone there was great, and I even came away with slightly negative impressions of some of them. I never became a regular Slate Star Codex reader.

On the train back, I talked to the two Yale students who came with me. One of them later became a very good friend of mine. I remember finding him just interesting, and wanting to spend more time talking to him. He was eager to talk to me, and share his knowledge and thoughts, and he made me feel included. If he hadn’t gone, I’m not sure the trip would have translated into me getting more involved with Yale EA, because I wouldn’t have felt as connected to anyone in it. But he clearly made an effort to get to know me, and that made all the difference in my involvement with EA.

All because I got in the van.

Why did I get in the van?

You may be wondering why I got in a van with strangers to go to an event, when I didn’t even know what the event was. If any one of the reasons below hadn’t been true, I wouldn’t have gotten in the van:

I had attended one Yale EA event before, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Off Road: support for EAs struggling at uni, published by technicalities on January 21, 2022 on The Effective Altruism Forum.

This project is supported by an EA Infrastructure Fund grant.

tl;dr: Executive dysfunction is common and can make uni intolerable. We want to help. Sign up to be an accountability partner now, or tell us in the comments what's needed.

I (Gavin) keep meeting young people who struggle enormously with university. I now personally know 9 EAs who suffered lasting mental health problems they associate with uni, or who are currently thinking of dropping out. (This is about 5% of all EAs I have spoken to intimately.) 4% of EAs taking the 2019 Survey reported “Some college, no degree”. 3% of EAs in the 2020 SSC survey reported having less than an associate degree as their completed education.

There's nothing inherently wrong with dropping out: uni isn't for everyone. But we (Damon and Gavin) think this implies a few problems:

Some EAs are on suboptimal trajectories because of poor fit with uni / other mental health problems.

There's a shortage of in-community coaching and mentorship, particularly for people with these problems / less obvious potential.

Dropping out entails shame or trauma - even though dropping out isn’t a strong disqualifier in many careers, including EA careers. Prospective suffering makes (1) much worse.

The above, and lack of attention to the above, harm community morale, even among those who don’t themselves have these problems.

Some people aren’t constituted to sit through college, or doubt the emotional cost-benefit ratio in their case. Some could make it through with less trauma, given a bit of help. So we’re doing something for people who struggle in these ways.

Gavin’s hunch was that executive dysfunction was the major neglected factor (distinct from well-known conditions like anxiety, depression, or scrupulosity). Damon (a professional therapist) came on board to dig into what that actually means, and we designed a survey to test the hunch. We did some small things on our small exploratory grant and have ideas for how to scale up.

What is executive dysfunction?

First things first; “executive dysfunction” is not a diagnosis. Executive functions are what govern our ability to plan actions, take those actions, maintain focus on them, adapt to changes, and more subtle steps between.

ADHD is a diagnosis that points to a cluster of common struggles with executive function: working memory, impulse control, and self monitoring. But there are plenty of other diagnoses that can impact one or more of those eight, and of course even things like lack of sleep, hunger, being irritated, disruptive environments, and other stressors can affect them.

So in general when we talk about executive dysfunction what we’re really pointing at is a symptom we witness when someone isn’t able to act on their desires, or on things they think they should do, or on things they think they should desire.

Which brings up the more philosophical question; what does it mean to “fail to act” on a desire? Does someone “have executive dysfunction” if they struggle to complete something they don’t want to do, but feel they have to? What about what they “want to want” to do, but don’t find interesting, even while they can still work on passion projects without issue? Or is it only executive dysfunction if they can’t bring themselves to work on something they feel a strong desire to do, in which case what does “strong desire” mean?

All this makes the question of whether someone struggles with executive dysfunction ill-posed. The better question is “in what domains or in what types of circumstances does someone struggle with executive dysfunction,” followed by narrowing down to which of their executive functions are the chokepoint. Organization? Task initiation? Emotion...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sharing Powerful AI Models, published by Alexis Carlier on January 21, 2022 on The AI Alignment Forum.

This is a linkpost for:

On the GovAI blog, Toby Shevlane (FHI) argues in favour of labs granting "structured access" to AI models.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implications of Civilizational Inadequacy (reviewing mazes/simulacra/etc), published by Raemon on January 21, 2022 on LessWrong.

A high level review, exploring multiple angles to look at the collection of 2020 posts on Moral Mazes, Simulacra, Coordination Theory, and Civilizational Adequacy.

I. What Makes For a Useful Review?

When reviewing LessWrong posts, fairly often my reaction is "Cool hypothesis bro, but, can we actually go out and verify it?". A lot of LessWrong posts explore interesting ideas, and those ideas match my intuitions. But lots of intuitive ideas turn out to be wrong, and sometimes people have different intuitions. It seemed good to have a step where we actually turned armchair philosophy into empirical claims.

A year ago, I began two attempts at investing some claims:

Eliezer's conjecture that Mathematicians were more law-abiding that non-mathematicians (controlling for IQ).

The overall Moral Mazes sequence.

Mathematicians and "Is this data worth the expense?"

Elizabeth and I made some attempt to run a Positly study on math / law-abidingness. We quickly ran into "Running a real, legit study is a pretty big time cost." (We ran one quick-and-dirty study that essentially boiled down to asking people 'are you good at math?' and 'do you do crimes?', which Eliezer specifically called out as Not A Good Approach in his post. We tried looking for datasets that might be relevant to our interests, but finding and sifting through them was hard).

Ultimately I faced the question: "is this question actually worth investing this effort in?" and I felt the answer was "probably not." It seemed vaguely informative, and I felt a nagging annoyance at publishing Local Validity As Key To Sanity and Civilization without having investigated it. But after a lot of reflection (on the timescale of years), I came to the belief that not every claim is worth investigating.

If I was worried about the epistemic integrity of the LessWrong books, a much cheaper approach would be to include some metadata about the epistemic status of each post, or suggestions of future work that signposted which areas warranted more investigation.

Moral Mazes, and "What are we actually trying to figure out?"

The Immoral Mazes sequence raised a lot of interesting hypotheses, which matched my intuitions/experience. But it seemed to me the next step was do form some actual predictions and go out and test them.

At the time, I was thinking about Mazes largely from the standpoint of "I want to build a rationalist/EA/longtermist ecosystem that accomplishes good at scale. How do we do that, without succumbing to Mazey pressures?". Example of questions I wanted to test were: "Do more layers of hierarchy reliably lead to more Mazeyness? How reliably? What observations do I actually expect if an org is 'mazey'? Does it help if an org is broken into smaller pieces or does that just shuffle around the problem? How do we evaluate middle managers?"

Testing these things would not be a cheap undertaking. But unlike the Mathematician example, it actually seemed important.

I went to chat with Oliver Habryka about it, and he noted (paraphrased) "For building an organization, I'm not sure it's actually all that useful to study how exactly things fail. I'd much rather see a bunch of case studies of how things succeeded. What types of orgs produce value, and how do they go about it?"

This was an interesting takeaway for me – when reviewing posts, the most important takeaways don't necessarily look like "evaluate specific claims". The important thing might be "figure out what decisions this post bears on, use the post as a springboard/pointer to help figure out what you actually need to know.

In my ideal world, there's funding and bandwidth to followup on lots of claims, fleshing out our understanding. Realistically, that ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is A case for the effectiveness of protest, published by James Ozden.

What I want this post to achieve: My main goal with this post is to start a discussion about the effectiveness of different forms of political advocacy. Specifically, whether, and how, social movement concepts such as nonviolent protest should be used for EA causes.

Disclaimer and epistemic certainty: This is a somewhat speculative post and I’m not fully confident on some of the numbers used for the cost-effectiveness estimates. I’ve been working for Extinction Rebellion and Animal Rebellion for the past two years, as well as studying social movement theory, so I will naturally bring in some degree of bias and motivated reasoning. I also think it’s important to note that due to concerns around The Sunrise Movement expressed here, I have significantly weakened my belief in the effectiveness of certain social movements.

Reading time: 30-60+ minutes. You can also read it in a Google Doc.

Summary

Social movements are broad alliances of people who are connected through their shared interest in social change. This research focuses on social movements that use civil resistance as a theory of change, as I believe this is under-represented within Effective Altruism (EA). Civil resistance can be defined as political action that relies on the use of nonviolent resistance by civil groups to challenge a particular power, force, policy or regime. In practice, this looks like nonviolent protests and direct action.

Extinction Rebellion (XR) has highlighted the potential for social movements using nonviolent protest to create positive societal change. However, there has been little quantitative analysis of the exact impact that XR or other social movements have had on shifting public opinion, creating policy change or, in this case, reducing carbon emissions. In this research project, I attempted to quantify the cost-effectiveness that XR's protests, and other activities, has had on reducing greenhouse gas emissions (GHG) and influencing government spending on climate-related activities. These findings suggest that XR has abated 16 tonnes of GHGs per pound spent on advocacy, using the median estimates for cost-effectiveness. Relative to the top Effective Altruist (EA) recommended climate change charity, Clean Air Task Force (CATF), this is more effective by a factor of 12x. If true, this indicates that nonviolent protests can be highly effective in achieving positive outcomes and social movement objectives. This leads to the conclusion that social movement theory should be a focus area for impact-focused researchers, advocates and philanthropists, to determine when these opportunities might arise and how to best utilise them.

Throughout this research, I argue for the following claims, which I believe to be strong:

Nonviolent protest is an effective tool to influence public opinion and policy around a certain issue.

Public opinion plays a significant factor in policy change

To date, Effective Altruists have devoted too little consideration to social movements and civil resistance.

I’m also arguing for the following claims, but I believe them to be weaker:

The most impactful Social Movement Organisations (SMOs) using nonviolent protest can be more cost-effective than existing EA-funded interventions. My cost-effectiveness analysis of Extinction Rebellion indicates that they were more cost-effective, by a factor of 0.4 - 32x, than current EA recommendations for tackling climate change, using a variety of metrics.

A two-person year research project studying the use of social movements and civil resistance for certain cause areas could discover more cost-effective interventions than those that already exist. I estimate there's a 30% likelihood of this happening.

We should allocate a greater proportion of funds towards early...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Most successful EA elevator pitches?, published by Ines on January 19, 2022 on The Effective Altruism Forum.

Has anyone done any surveys, focus groups, etc on the most effective way to pitch EA in a couple of sentences? I would roughly define "most effective" as whichever most frequently sparks curiosity or enthusiasm in the recipient while maintaining accuracy. I often run into the problem of EA coming up in casual conversation and not knowing exactly how to explain what it is, and I know many others run into this problem as well. A large study on this could be very valuable as this little spiel is among the key make-or-break factors in someone wanting to know more about EA (and possibly go on to become involved) or not, and some pitches may be orders of magnitude better than others (of course, the optimal one for any given person or group will vary, but in general). Seems like a no-brainer that at the very least organizers and community builders should be optimizing this.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.