Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against “longtermist” as an identity, published by Lizka on May 13, 2022 on The Effective Altruism Forum.

This is mostly addressing people who care a lot about improving the long-term future and helping life continue for a long time, and who might be tempted to call themselves “longtermist.”

There have been discussions about how “effective altruist” shouldn’t be an identity and some defense of EA-as-identity. (I also think I’ve seen similar discussions about “longtermists” but don’t remember where.) In general, there has been a lot of good content on the effect of identities on truth-seeking conversation (see Scout Mindset or “Keep Your Identity Small”).

But are there actual harms of identifying as a “longtermist”? I describe two in this post; it can make it harder to change your mind based on new information, and it can make your conversations and beliefs more confused by adding aspects of the group identity that you’d otherwise not have adopted as part of your individual identity.

(0) What is “longtermism”?

When people say “I’m a longtermist,” they mean something like: “I subscribe to the moral-philosophical theory of longtermism.’”

So, what is “longtermism?” The definition given in various places is “the view that positively influencing the long-term future is a key moral priority of our time.” In practice, this often relies on certain moral and empirical beliefs about the world:

Believing that future beings are morally relevant — this belief is needed in order to argue that we should put resources to helping them if we can, even if those efforts trade off against resources that could go to beings alive today (a moral belief)

Not putting a significant “pure temporal discount rate” on the moral value of future beings (a person in 1000 years is “worth” basically as much as a person today) (a moral belief)

Believing that the future is big — that the value of the future can be enormous, and thus the scope of the issue is huge (an empirical belief)

And thinking that it’s actually possible to help future beings in non-negligible ways in expectation (an empirical belief)

It’s worth pointing out that most people (even those who wouldn’t call themselves “longtermist”) care about future beings. Some people disagree on (2) or (3). And lots of people in effective altruism who disagree with longtermism, I think, disagree primarily with (4): the feasibility of predictably helping the future.

Importantly, I think that while your position on 1-2 can be significantly dependent on strongly felt beliefs, 3-4 are more based on facts about the world: empirical data and arguments that can be refuted.

(1) Calling yourself “longtermist” bakes empirical or refutable claims into an identity, making it harder to course-correct if you later find out you’re wrong.

Rewind to 2012 and pretend that you’re an American who’s realized that, if you want to improve the lives of (current) people, you should probably be aiming your efforts at helping people in poorer countries, not the US. You’d probably not call yourself a "poor-country-ist." Instead, you might go for “effective altruist” and treat the conclusion that the most effective interventions are aimed at people in developing countries as an empirical conclusion given certain assumptions (like impartiality over geographic location). If it turns out that the new most effective intervention would be to help Americans, you can pivot to that without sacrificing or fighting your identity.

Some of the earliest and most influential proponents of longtermism agree that longtermism is not the for-sure-correct approach/belief/philosophy.

For instance, Holden Karnofsky writes:

There are arguments that our ethical decisions should be dominated by concern for ensuring that as many people as possible will someday get to exist. I really go back and forth on ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Time in Agent Models, published by johnswentworth on May 13, 2022 on The AI Alignment Forum.

When programming distributed systems, we always have many computations running in parallel. Our servers handle multiple requests in parallel, perform read and write operations on the database in parallel, etc.

The prototypical headaches of distributed programming involve multiple processes running in parallel, each performing multiple read/write operations on the same database fields. Maybe some database field says “foo”, and process 1 overwrites it with “bar”. Process 2 reads the field - depending on the timing, it may see either “foo” or “bar”. Then process 2 does some computation and writes another field - for instance, maybe it sees “foo” and writes {“most_recent_value”: “foo”} to a cache. Meanwhile, process 1 overwrote “foo” with “bar”, so it also overwrites the cache with {“most_recent_value”: “bar”}. But these two processes are running in parallel, so these operations could happen in any order - including interleaving. For instance, the order could be:

Process 2 reads “foo”

Process 1 overwrites “foo” with “bar”

Process 1 overwrites the cache with {“most_recent_value”: “bar”}

Process 2 overwrites the cache with {“most_recent_value”: “foo”}

. and now the cached value no longer matches the value in the database; our cache is broken.

One of the main heuristics for thinking about this sort of problem in distributed programming is: there is no synchronous time. What does that mean?

Well, in programming we often picture a “state-update” model: the system has some state, and at each timestep the state is updated. The update rule is a well-defined function of the state; every update happens at a well-defined time. This is how each of the individual processes works in our example: each executes two steps in a well-defined order, and each step changes the state of the system

But with multiple processes in parallel, this state-update model no longer works. In our example, we can diagram our two processes like this:

Each process has its own internal “time”: the database read/write happens first, and the cache overwrite happens second. But between processes, there is no guaranteed time-ordering. For instance, the first step of process 1 could happen before all of process 2, in between the steps of process 2, or after all of process 2.

We cannot accurately represent this system as executing along one single time-dimension. Proof:

Step 1 of process 1 is not guaranteed to happen either before or after step 1 of process 2; at best we could represent them as happening “at the same time”

Step 2 of process 1 is also not guaranteed to happen either before or after step 1 of process 2; at best we could represent them as happening “at the same time”

. but step 2 of process 1 is unambiguously after step 1 of process 1 in time, so the two steps can’t happen at the same time.

In order to accurately represent this sort of thing, it has to be possible for one step to be unambiguously after another, even though both of them are neither before nor after some third step.

The “most general” data structure to represent such a relationship is not a one-dimension “timeline” (i.e. total order), but rather a directed acyclic graph (i.e. partial order). That’s how time works in distributed systems: it’s a partial order, not a total order. A DAG, not a timeline. That DAG goes by many different names - including computation DAG, computation circuit, or causal model.

Beyond Distributed Programming

The same basic idea carries over to distributed systems more generally - i.e. any system physically spread out in space, with lots of different stuff going on in parallel. In a distributed system, “time” is a partial order, not a total order.

In the context of embedded agents: we want to model agenty systems ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deferring, published by Owen Cotton-Barratt on May 12, 2022 on The Effective Altruism Forum.

[meta: my attempt to find a good big-picture framing for a topic that's clearly important but I am not convinced we're nailing]

Deferring is when you adopt someone else's view on a question over your own independent view (or instead of taking the time to form an independent view). You can defer on questions of fact or questions of what to do. You might defer because you think they know better (epistemic deferring), or because there is a formal or social expectation that you should go along with their view (deferring to authority).

Both types of deferring are important — epistemic deferring lets people borrow the fruits of knowledge; deferring to authority enables strong coordination. But they are two-edged. Deferring can mean that you get less chance to test out your own views, so developing mastery is slower. Deferring to the wrong people can be straightforwardly bad. And when someone defers without everyone understanding that's what's happening, it can cause issues. Similarly, unacknowledged expectations of deferral from others can cause problems. We should therefore learn when and how to defer, when not to, and how to be explicit about what we're doing.

Why deferring is useful

Epistemic deferring

Epistemic deferring is giving more weight to someone else's view than your own because you think they're in a position to know better. The opposite of epistemic deferring is holding one's own view.

Examples:

"You've been to this town before; where's the best place to get coffee?"

"My doctor/lawyer says this is a common situation, and the right thing to do is ..."

"A lot of smart folks seem to think AI risk is a big deal; it sounds batshit to me, but I guess I'll look into it more"

The case for epistemic deferring is simple: for most questions, we can identify someone (or some institution or group of people) whose judgement on the question would — if they were possessed of the facts we knew — be better than our own. So to the extent that

(A) We want to optimize for accurate judgements above all else, &

(B) We are willing to make the investment to uncover that better judgement,

deferring will be correct.

Partial deferring

The degree to which (A) and (B) hold will vary with circumstance. It will frequently be the case that they partially hold; in this case it may be appropriate to partially defer, e.g.

“I’m torn between whether to take job X or job Y. On my view job X seems better. When I talk to my friends and family they overwhelmingly think job Y sounds better; maybe they’re seeing something I’m not. If I thought it was a close call anyway this might be enough to tip me over, but it won’t change my mind if my preference for X was clear.”

Deferring to authority

Deferring to authority is adopting someone else's view because of a social contract to do so. Often deferring to authority happens on questions of what should be done — e.g. "I'm going to put this fire alarm up because [my boss / my client / the law] tells me to", or “I’m helping my friend cook dinner, so I’ll cut the carrots the way they want, even though I think this other way is better”. The opposite of deferring to authority is acting on one's own conscience.

Deferring to authority — and the reasonable expectation of such deferring — enables groups of people to coordinate more effectively. Militaries rely on it, but so do most projects (large and small, but especially large). It's unreasonable to expect that everyone working on a large software project will have exactly the same views over the key top-level design choices, but it's better if there's some voice that can speak authoritatively, so everyone can work on that basis. If we collectively want to be able to undertake large ambitious projects, we’ll likely need to ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Agency As a Natural Abstraction, published by Thane Ruthenis on May 13, 2022 on The AI Alignment Forum.

Epistemic status: Speculative attempt to synthesize findings from several distinct approaches to AI theory.

Disclaimer: The first three sections summarize some of Chris Olah's work on interpretability and John Wentworth's Natural Abstractions Hypothesis, then attempt to draw connections between them. If you're already familiar with these subjects, you can probably skip all three parts.

Short summary: When modelling a vast environment where simple rules result in very complex emergent rules/behaviors (math, physics...), it's computationally efficient to build high-level abstract models of this environment. Basic objects in such high-level models often behave very unlike basic low-level objects, requiring entirely different heuristics and strategies. If the environment is so complex you build many such models, it's computationally efficient to go meta, and build a higher-level abstract model of building and navigating arbitrary world-models. This higher-level model necessarily includes the notions of optimization and goal-orientedness, meaning that mesa-optimization is the natural answer to any "sufficiently difficult" training objective. All of this has various degrees of theoretical, empirical, and informal support.

1. The Universality Hypothesis

One of the foundations of Chis Olah's approach to mechanistic interpretability is the Universality Hypothesis. It states that neural networks are subject to convergence — that they would learn to look for similar patterns in the training data, and would chain up the processing of these patterns in similar ways.

The prime example of this effect are CNNs. If trained on natural images (even from different datasets), the first convolution layer reliably learns Gabor filters and color-contrast detectors, and later layers show some convergence as well:

It's telling that these features seem to make sense to us, as well — that at least one type of biological neural network also learns similar features. (Gabor filters, for example, were known long before modern ML models.) It's the main reason to feel optimistic about interpretability at all — it's plausible that the incomprehensible-looking results of matrix multiplications will turn out to be not so incomprehensible, after all.

It's telling when universality doesn't hold, as well.

Understanding RL Vision attempts to interpret an agent trained to play CoinRun, a simple platformer game. CoinRun's levels are procedurally generated, could contain deadly obstacles in the form of buzzsaws and various critters, and require the player to make their way to a coin.

Attempting to use feature visualization on the agent's early convolutional layers produces complete gibberish, lacking even Gabor filters:

It's nonetheless possible to uncover a few comprehensible activation patterns via the use of different techniques:

The agent learns to associate buzzsaws and enemies with decreased chances of successfully completing a level, and could be seen to pick out coins and progression-relevant level geometry.

All of these comprehensible features, however, reside on the third convolutional layer. None of the other four convolutional layers, or the two fully-connected layers, contain anything that makes sense. The authors note the following:

Interestingly, the level of abstraction at which [the third] layer operates – finding the locations of various in-game objects – is exactly the level at which CoinRun levels are randomized using procedural generation. Furthermore, we found that training on many randomized levels was essential for us to be able to find any interpretable features at all.

At this point, they coin the Diversity Hypothesis:

Interpretable features tend to arise (at a given level of abstra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Tech company singularities", and steering them to reduce x-risk, published by Andrew Critch on May 13, 2022 on LessWrong.

The purpose of this post (also available on the EA Forum) is to share an alternative notion of “singularity” that I’ve found useful in timelining/forecasting.

A fully general tech company is a technology company with the ability to become a world-leader in essentially any industry sector, given the choice to do so — in the form of agreement among its Board and CEO — with around one year of effort following the choice.

Notice here that I’m focusing on a company’s ability to do anything another company can do, rather than an AI system's ability to do anything a human can do. Here, I’m also focusing on what the company can do if it chooses rather than what it actually ends up choosing to do. If a company has these capabilities and chooses not to use them — for example, to avoid heavy regulatory scrutiny or risks to public health and safety — it still qualifies as a fully general tech company.

This notion can be contrasted with the following:

Artificial general intelligence (AGI) refers to cognitive capabilities fully generalizing those of humans.

An autonomous AGI (AAGI) is an autonomous artificial agent with the ability to do essentially anything a human can do, given the choice to do so — in the form of an autonomously/internally determined directive — and an amount of time less than or equal to that needed by a human.

Now, consider the following two types of phase changes in tech progress:

A tech company singularity is a transition of a technology company into a fully general tech company. This could be enabled by safe AGI (almost certainly not AAGI, which is unsafe), or it could be prevented by unsafe AGI destroying the company or the world.

An AI singularity is a transition from having merely narrow AI technology to having AGI technology.

I think the tech company singularity concept, or some variant of it, is important for societal planning, and I’ve written predictions about it before, here:

2021-07-21 — prediction that a tech company singularity will occur between 2030 and 2035

2022-04-11 — updated prediction that a tech company singularity will occur between 2027 and 2033.

A tech company singularity as a point of coordination and leverage

The reason I like this concept is that it gives an important point of coordination and leverage that is not AGI, but which interacts in important ways with AGI. Observe that a tech company singularity could arrive

before AGI, and could play a role in

preventing AAGI, e.g., through supporting and enabling regulation;

enabling AGI but not AAGI, such as if tech companies remain focussed on providing useful/controllable products (e.g., PaLM, DALL-E);

enabling AAGI, such as if tech companies allow experiments training agents to fight and outthink each other to survive.

after a tech company singularity, such as if the tech company develops safe AGI, but not AAGI (which is hard to control, doesn't enable the tech company to do stuff, and might just destroy it).

Points (1a) and (1b) are, I think, humanity’s best chance for survival. Moreover, I think there is some chance that the first tech company singularity could come before the first AI singularity, if tech companies remain sufficiently oriented on building systems that are intended to be useful/usable, rather than systems intended to be flashy/scary.

How to steer tech company singularities?

The above suggests an intervention point for reducing existential risk: convincing a mix of

scientists

regulators

investors, and

the public

. to shame tech companies for building useless/flashy systems (e.g., autonomous agents trained in evolution-like environments to exhibit survival-oriented intelligence), so they remain focussed on building usable/useful systems (e.g., D...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An observation about Hubinger et al.'s framework for learned optimization, published by Spencer Becker-Kahn on May 13, 2022 on The AI Alignment Forum.

The observations I make here have little consequence from the point of view of solving the alignment problem. If anything, they merely highlight the essential nature of the inner alignment problem. I will reject the idea that robust alignment, in the sense described in Risks From Learned Optimization, is possible at all. And I therefore also reject the related idea of 'internalization of the base objective', i.e. I do not think it is possible for a mesa-objective to "agree" with a base-objective or for a mesa-objective function to be “adjusted towards the base objective function to the point where it is robustly aligned.” I claim that whenever a learned algorithm is performing optimization, one needs to accept that an objective which one did not explicitly design is being pursued. At present, I refrain from attempting to propose my own adjustments to the framework, or to build on the existing literature or to develop my own theory.

I am certainly not against doing any of those things, but they are things to possibly be pursued later; none of them is the purpose of this post.

To make my main point, I will introduce only a bare minimum of mathematical notation. We will show that a mesa-objective always has a different type signature to a base objective and that the default assumption ought to be that there is no way to compare them in general and certainly no general way to interpret what it means for them to ‘agree’. Suppose that an optimizer is searching through a space S of systems. At this time, I do not want to attempt to unpack what it means to 'search', but, naively, we can imagine that there is an objective function f :S→R, which determines something that we might call the 'search criterion'. The idea of course is that the optimizer is a system that is 'searching' through the set S and judging different points according to the criterion that higher values of f are better.

In the background, there is some 'task' and naively we can think of this as being represented by a 'task space' X which consists of all of the different possible 'presentations' or 'instances' of the task. For example, perhaps the task is choosing the next move in a game of Go or the next action in a real-time strategy video game. In these examples, a given x∈X would represent a board position in Go, say, or a single snapshot of the game-state in the video game. Then, in general, given x∈X and s∈S, we can think that s(x) is the output of s on the task instance x or the action taken by s when presented with x (i.e. s(x) denotes the next board move in Go or the next action to be taken in the video game). So each element of S defines a map from the task space X to some kind of output space or space of possible actions, which we need not notate.

Now, it is possible that there exists m∈S which works in the following way: Whenever the output of m on an instance x of the task needs to be evaluated, i.e. whenever m(x) is computed, what happens is that m searches over another search space Σ and looks for elements that score highly according to some other objective function g:Σ→R. Whenever this is the case, we say that such an m∈S is a mesa-optimizer and that the original optimizer - the one that searches over S - is the base optimizer. Notice that in some way, elements of Σ must in turn correspond to outputs/actions, because given some x, the mesa-optimizer m conducts a search over Σ to determine what output m(x) is, but that is all just part of the internal workings of m and we need not 'know' or notate how this correspondence works.

In Risks From Learned Optimization, Hubinger et al. write:

In such a case, we will use base objective to refer to whateve...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Results from the First Decade Review, published by Lizka on May 13, 2022 on The Effective Altruism Forum.

Tl;dr:

The Decade Review happened: Forum users voted on which posts from the first decade of effective altruism they found most useful and important and reviewed the posts to explain what they appreciated.

We’re awarding $12,500 in prizes to authors and reviewers.

The distribution of prizes to authors was based on your votes, the results of which are linked.

I’m using this post to highlight some aspects of the winning posts and reviews. I encourage more discussion in the comments (and in new posts!).

Prizes for posts

We’re awarding a total of $10,000 to the top posts (and $2,500 for the top reviews). Note that some authors chose to donate their prizes.

Summary

We’re awarding $10,000 to authors of the posts, broken down as follows.

$1,500 to Hauke Hillebrandt and John G. Halstead ($750 each) for Growth and the case against randomista development

$1000 each to

Helen for Effective Altruism is a Question (not an ideology)

Jai for 500 Million, But Not A Single One More

Greg Lewis for Beware surprising and suspicious convergence

Nate Soares for On Caring

Luisa Rodriguez for two posts(!) : What is the likelihood that civilizational collapse would directly lead to human extinction (within decades)? and How bad would nuclear winter caused by a US-Russia nuclear exchange be?

$500 each to

David Althaus and Tobias Baumann ($250 each) for Reducing long-term risks from malevolent actors

Will MacAskill for Are we living at the most influential time in history?

Holly Elmore for We are in triage every second of every day

“EA applicant” for After one year of applying for EA jobs: It is really, really hard to get hired by an EA organisation

Brian Tomasik for Why Charities Usually Don't Differ Astronomically in Expected Cost-Effectiveness

Julia Wise for You have more than one goal, and that's fine

Peter Singer for Peter Singer – Famine, Affluence, and Morality

First prize ($1500): Growth and the case against randomista development by Hauke Hillebrandt and John Halstead

Randomista development is “an approach to development economics [that focuses on] interventions which can be tested by randomised controlled trials (RCTs).” This is popular in effective altruism, but, as this post argues, might systematically undervalue or miss the interventions that have been responsible for some of the biggest improvements ever in quality of life. In particular, the authors discuss the importance of economic growth for human welfare and argue that the welfare gains from increasing GDP per capita in a country are so large that they outweigh the benefits of randomista development programs by orders of magnitude. As a result, they argue, researchers and grantmakers should spend much more time looking for interventions that can promote growth. The post also discusses objections and explores the limitations of GDP/growth, but emphasizes that this topic is under-prioritized by people in effective altruism.

I like a lot of things about this post. I had to print the post out and read it carefully (pen, highlighters, and all), and don’t claim expertise in the subjects involved, but I understand enough to see that it’s making some important and true points (although you should also read the comments on the post for a discussion on potential weaknesses and more disputable claims).

I think more people should explore the value and tractability of economic growth, and more people might want to read this post as an introduction to that subject. Beyond that, it’s also a good example to follow if you’re considering writing or researching something. So I’m glad this post has won the first prize.

Or, as Maxime CdS put it,

[...] this post makes a very good point in a very important conversation, namely that we can do b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind is hiring for the Scalable Alignment and Alignment Teams, published by Rohin Shah on May 13, 2022 on The Effective Altruism Forum.

We are hiring for several roles in the Scalable Alignment and Alignment Teams at DeepMind, two of the subteams of DeepMind Technical AGI Safety trying to make artificial general intelligence go well. In brief,

The Alignment Team investigates how to avoid failures of intent alignment, operationalized as a situation in which an AI system knowingly acts against the wishes of its designers. Alignment is hiring for Research Scientist and Research Engineer positions.

The Scalable Alignment Team (SAT) works to make highly capable agents do what humans want, even when it is difficult for humans to know what that is. This means we want to remove subtle biases, factual errors, or deceptive behaviour even if they would normally go unnoticed by humans, whether due to reasoning failures or biases in humans or due to very capable behaviour by the agents. SAT is hiring for Research Scientist - Machine Learning, Research Scientist - Cognitive Science, Research Engineer, and Software Engineer positions.

We elaborate on the problem breakdown between Alignment and Scalable Alignment next, and discuss details of the various positions.

“Alignment” vs “Scalable Alignment”

Very roughly, the split between Alignment and Scalable Alignment reflects the following decomposition:

Generate approaches to AI alignment – Alignment Team

Make those approaches scale – Scalable Alignment Team

In practice, this means the Alignment Team has many small projects going on simultaneously, reflecting a portfolio-based approach, while the Scalable Alignment Team has fewer, more focused projects aimed at scaling the most promising approaches to the strongest models available.

Scalable Alignment’s current approach: make AI critique itself

Imagine a default approach to building AI agents that do what humans want:

Pretrain on a task like “predict text from the internet”, producing a highly capable model such as Chinchilla or Flamingo.

Fine-tune into an agent that does useful tasks, as evaluated by human judgements.

There are several ways this could go wrong:

Humans are unreliable: The human judgements we train against could be flawed: we could miss subtle factual errors, use biased reasoning, or have insufficient context to evaluate the task.

The agent’s reasoning could be hidden: We want to know not just what the system is doing but why, both because that might reveal something about what that we don’t like, and because we expect good reasoning to better generalize to other situations.

Even if the agent is reasoning well, it could fail in other situations: Even if the reasoning is correct this time, the AI could fail to generalize correctly to other situations.

Our current plan to address these problem is (in part):

Give humans help in supervising strong agents: On the human side, provide channels for oversight and advice from peers, experts in various domains, and broader society. On the ML side, agents should explain their behaviour and reasoning, argue against themselves when wrong, and cite relevant evidence.

Align explanations with the true reasoning process of the agent: Ensure that agent’s are able and incentivized to show their reasoning to human supervisors, either by making reasoning explicit if possible or via methods for interpretability and eliciting latent knowledge.

Red team models to exhibit failure modes that don’t occur in normal use

We believe none of these pieces are sufficient by themselves:

(1) without (2) can be rationalization, where an agent decides what to do and produces an explanation after the fact that justifies its answer.

(2) without (1) doesn’t scale: The full reasoning trace of the agent might be enormous, it might be terabytes of data even wi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind is hiring for the Scalable Alignment and Alignment Teams, published by Rohin Shah on May 13, 2022 on The AI Alignment Forum.

We are hiring for several roles in the Scalable Alignment and Alignment Teams at DeepMind, two of the subteams of DeepMind Technical AGI Safety trying to make artificial general intelligence go well. In brief,

The Alignment Team investigates how to avoid failures of intent alignment, operationalized as a situation in which an AI system knowingly acts against the wishes of its designers. Alignment is hiring for Research Scientist and Research Engineer positions.

The Scalable Alignment Team (SAT) works to make highly capable agents do what humans want, even when it is difficult for humans to know what that is. This means we want to remove subtle biases, factual errors, or deceptive behaviour even if they would normally go unnoticed by humans, whether due to reasoning failures or biases in humans or due to very capable behaviour by the agents. SAT is hiring for Research Scientist - Machine Learning, Research Scientist - Cognitive Science, Research Engineer, and Software Engineer positions.

We elaborate on the problem breakdown between Alignment and Scalable Alignment next, and discuss details of the various positions.

“Alignment” vs “Scalable Alignment”

Very roughly, the split between Alignment and Scalable Alignment reflects the following decomposition:

Generate approaches to AI alignment – Alignment Team

Make those approaches scale – Scalable Alignment Team

In practice, this means the Alignment Team has many small projects going on simultaneously, reflecting a portfolio-based approach, while the Scalable Alignment Team has fewer, more focused projects aimed at scaling the most promising approaches to the strongest models available.

Scalable Alignment’s current approach: make AI critique itself

Imagine a default approach to building AI agents that do what humans want:

Pretrain on a task like “predict text from the internet”, producing a highly capable model such as Chinchilla or Flamingo.

Fine-tune into an agent that does useful tasks, as evaluated by human judgements.

There are several ways this could go wrong:

Humans are unreliable: The human judgements we train against could be flawed: we could miss subtle factual errors, use biased reasoning, or have insufficient context to evaluate the task.

The agent’s reasoning could be hidden: We want to know not just what the system is doing but why, both because that might reveal something about what that we don’t like, and because we expect good reasoning to better generalize to other situations.

Even if the agent is reasoning well, it could fail in other situations: Even if the reasoning is correct this time, the AI could fail to generalize correctly to other situations.

Our current plan to address these problem is (in part):

Give humans help in supervising strong agents: On the human side, provide channels for oversight and advice from peers, experts in various domains, and broader society. On the ML side, agents should explain their behaviour and reasoning, argue against themselves when wrong, and cite relevant evidence.

Align explanations with the true reasoning process of the agent: Ensure that agent’s are able and incentivized to show their reasoning to human supervisors, either by making reasoning explicit if possible or via methods for interpretability and eliciting latent knowledge.

Red team models to exhibit failure modes that don’t occur in normal use

We believe none of these pieces are sufficient by themselves:

(1) without (2) can be rationalization, where an agent decides what to do and produces an explanation after the fact that justifies its answer.

(2) without (1) doesn’t scale: The full reasoning trace of the agent might be enormous, it might be terabytes of data even with com...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Look Out The Window, published by ThomasWoodside on May 12, 2022 on The Effective Altruism Forum.

I wrote the following for a Yale EA leadership strategy meeting. It's optimized for speaking, not writing, so it might read weirdly. It also has some Yale EA specific bits. People seemed to value it, and for various reasons now seems like a good time for me to post it.

Look out the window. That church was built in 1638.

Back then, smallpox was wiping out an entire native population. Slavery was spreading to the Americas. Women had no rights at all here. When most people considered the future, they considered merely their children. Empires colonized lands that were not theirs and subjected their people. Animals: they were just tools.

We killed smallpox, but COVID kills millions. We don’t have mass slavery, but we have racism, and there is still slavery. Women have rights, but we have sexism, and in some places women still have no rights. People barely consider their great-great grandchildren. Here we treat cancer but there we do not even treat malaria. Animals: they are still just tools.

If you hate suffering, you don’t have to look far. Yes, you can look at the parents of children dead from malaria. Or you can look out the window. Suffering is far too present in our world still.

If you love happiness, you don’t have to look far. You can watch me eating beets. Or you can look out the window. Happiness is everywhere in our world.

If you care about justice, virtue, knowledge, beauty, you don’t have to look far. Look out the window. These things and their opposites are everywhere in our world.

This window is like many other windows all over the world. Outside of some, there may only appear happiness, and others, suffering. But most are more like this window, somewhere in between the extremes. Even in the happiest idyllic landscape predators devour prey. Even in the most sad place, people still admire the moon sometimes.

We have a chance to change not just this window, but every window now, and every window in the future. There are arguments for why this could be the most important century to do just that. Our ability to get things done is exploding: we have people and we have money. Pivotal technologies, like synthetic biology and AI, are being developed as we speak. What’s our role?

We are part of a ragtag team of people who try to care about everyone and everything that matters. We are the first true attempt at applied impartial good maximization under these terms. If that’s not the most important thing you could be doing, I don’t know what is.

Now, that doesn’t guarantee that any of what we do here at Yale EA matters to any of this. I think it does. I see our alumni working on stopping pandemics, researching population ethics, assisting longtermism’s greatest thought leaders, doing high-leverage global development work, and starting successful groups at dozens of other universities, and I think: this group did that. And I think we can do even more.

Now, maybe you disagree. Maybe you think what you’re personally doing is not the best use of your time. I implore you: don’t do it! If there is something else that’s the best use of your time, go do that! If it’s a YEA thing, argue for why it’s the best use of your time, and go do it. If it’s not a YEA thing, don’t feel stuck here. Go do the more impactful thing!

I don’t care about Yale Effective Altruism, not in the way I care about you or my family or my friends or even my houseplants. Those things I care about in themselves, separately from how they maximize good. YEA, I don’t. I care about what’s out the window, and that’s why I care about this group.

Think about what’s out the window, not just this window but every window, and go make it better.

Thanks for listening. To help us out with The Nonlinear Library or to learn mo...