Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When to get off the train to crazy town?, published by mariushobbhahn on November 22, 2021 on The Effective Altruism Forum.

In her appearance on the 80k podcast, Ajeya Cotra introduces the concept of the train to crazy town.

Ajeya Cotra: And so when the philosopher takes you to a very weird unintuitive place — and, furthermore, wants you to give up all of the other goals that on other ways of thinking about the world that aren’t philosophical seem like they’re worth pursuing — they’re just like, stop. I sometimes think of it as a train going to crazy town, and the near-termist side is like, I’m going to get off the train before we get to the point where all we’re focusing on is existential risk because of the astronomical waste argument. And then the longtermist side stays on the train, and there may be further stops.

This analogy immediately resonated with me and I still like it a lot. In this post, I want to use this metaphor to clarify a couple of problems that I often encounter regarding career choice and other moral considerations.

Feel free to add metaphors in the comments.

The train to crazy town goes every five minutes:

Getting off the train is not a final decision. You can always hop back on. Clearly, there is some opportunity cost and a bit of lost time but in the broad scheme of things, there is still lots of room for change in your moral views. Especially when it comes to career choice, some people I talked to definitely take path dependencies too seriously. I had conversations along the lines of “I already did a Bachelor’s in Biology and just started a Master’s in Nanotech, surely it’s too late for me to pivot to AI safety”. To which my response is “You’re 22, if you really want to go into AI safety, you can easily switch”.

The return train goes every five minutes too:

If your current stop feels a bit too close to crazy town, you can always go back. The example from above also applies in the other direction. In general, I feel like people overestimate the importance of their path dependencies rather than updating them accordingly when their moral views change. Often, some years down the line, they can’t live with the discrepancy between their moral beliefs and lived actions anymore and change careers even later rather than cutting their losses early.

You can ride the train (nearly) for free:

There are a ton of ways in which you can do roundtrips to the previous or next town without having to get off the train. Obviously, there is a ton of stuff online such as 80K, Holden Karnofsky’s Aptitudes post, or probablygood, and many people use these resources already. However, I think people still don’t talk enough to other EAs about these questions. The amount of knowledge and experiences that you get from one weekend of 1on1s at EAG or EAGx is just insanely cheap and valuable compared to most reading or thinking that you can do on your own.

You are the conductor:

You control the speed at which your train rides. There is no reason to ride at an uncomfortably slow or fast paste if there is no reason for it.

More experienced people tend to ride longer:

What people intuitively think of as a “crazy view” depends on how long they consider themselves an EA, in my experience. I guess to my previous self from 6 years ago, some of my current beliefs would have seemed crazy.

More people are moving to further towns:

While the movement is growing a lot and therefore all towns become more populous, it seems like some further towns are getting disproportionately more citizens. Longtermism, AI safety, Biosecurity, and so on certainly were less mainstream 5 years ago than they are now.

There is no crazy town:

Nobody has found crazy town yet. Some people got on the first train went full gas no breaks and still haven’t reached it yet. I talked to someone with this mentality at EA...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When to get off the train to crazy town?, published by mariushobbhahn on November 22, 2021 on The Effective Altruism Forum.

In her appearance on the 80k podcast, Ajeya Cotra introduces the concept of the train to crazy town.

Ajeya Cotra: And so when the philosopher takes you to a very weird unintuitive place — and, furthermore, wants you to give up all of the other goals that on other ways of thinking about the world that aren’t philosophical seem like they’re worth pursuing — they’re just like, stop. I sometimes think of it as a train going to crazy town, and the near-termist side is like, I’m going to get off the train before we get to the point where all we’re focusing on is existential risk because of the astronomical waste argument. And then the longtermist side stays on the train, and there may be further stops.

This analogy immediately resonated with me and I still like it a lot. In this post, I want to use this metaphor to clarify a couple of problems that I often encounter regarding career choice and other moral considerations.

Feel free to add metaphors in the comments.

The train to crazy town goes every five minutes:

Getting off the train is not a final decision. You can always hop back on. Clearly, there is some opportunity cost and a bit of lost time but in the broad scheme of things, there is still lots of room for change in your moral views. Especially when it comes to career choice, some people I talked to definitely take path dependencies too seriously. I had conversations along the lines of “I already did a Bachelor’s in Biology and just started a Master’s in Nanotech, surely it’s too late for me to pivot to AI safety”. To which my response is “You’re 22, if you really want to go into AI safety, you can easily switch”.

The return train goes every five minutes too:

If your current stop feels a bit too close to crazy town, you can always go back. The example from above also applies in the other direction. In general, I feel like people overestimate the importance of their path dependencies rather than updating them accordingly when their moral views change. Often, some years down the line, they can’t live with the discrepancy between their moral beliefs and lived actions anymore and change careers even later rather than cutting their losses early.

You can ride the train (nearly) for free:

There are a ton of ways in which you can do roundtrips to the previous or next town without having to get off the train. Obviously, there is a ton of stuff online such as 80K, Holden Karnofsky’s Aptitudes post, or probablygood, and many people use these resources already. However, I think people still don’t talk enough to other EAs about these questions. The amount of knowledge and experiences that you get from one weekend of 1on1s at EAG or EAGx is just insanely cheap and valuable compared to most reading or thinking that you can do on your own.

You are the conductor:

You control the speed at which your train rides. There is no reason to ride at an uncomfortably slow or fast paste if there is no reason for it.

More experienced people tend to ride longer:

What people intuitively think of as a “crazy view” depends on how long they consider themselves an EA, in my experience. I guess to my previous self from 6 years ago, some of my current beliefs would have seemed crazy.

More people are moving to further towns:

While the movement is growing a lot and therefore all towns become more populous, it seems like some further towns are getting disproportionately more citizens. Longtermism, AI safety, Biosecurity, and so on certainly were less mainstream 5 years ago than they are now.

There is no crazy town:

Nobody has found crazy town yet. Some people got on the first train went full gas no breaks and still haven’t reached it yet. I talked to someone with this mentality at EA...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Needs Great Engineers, published by Andy Jones on November 23, 2021 on The AI Alignment Forum.

Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.

I work for Anthropic, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.

I've spoken to several other AI safety research organisations who feel the same.

Why engineers?

May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.

As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.

In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.

What kind of engineers?

Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are

experience with distributed systems

experience with numerical systems

caring about, and thinking a lot about, about AI safety

comfortable reading contemporary ML research papers

expertise in security, infrastructure, data, numerics, social science, or one of a dozen other hard-to-find specialities.

This is not a requirements list though. Based on the people working here already, 'great software engineer' and 'easy to get on with' are hard requirements, but the things in the list above are very much nice-to-haves, with several folks having just one or none of them.

Right now our job listings are bucketed into 'security engineer', 'infrastructure engineer', 'research engineer' and the like because these are the noun phrases that a lot of the people we like identify themselves with. But what we're actually most concerned about are generally-great software engineers who - ideally - have some extra bit of deep experience that we lack.

How does engineering compare to research?

At Anthropic there is no hard distinction between researchers and engineers. Some other organisations retain the distinction, but the increasing reliance of research on substantial, custom infrastructure is dissolving the boundary at every industrial lab I'm familiar with.

This might be hard to believe. I think the archetypal research-and-engineering organisation is one where the researchers come up with the fun prototypes, and then toss them over the wall to the engineers to clean up and implement. I think the archetype is common enough that it dissuades a lot of engineers from applying to engineering roles, instead applying to research...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Needs Great Engineers, published by Andy Jones on November 23, 2021 on The AI Alignment Forum.

Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.

I work for Anthropic, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.

I've spoken to several other AI safety research organisations who feel the same.

Why engineers?

May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.

As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.

In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.

What kind of engineers?

Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are

experience with distributed systems

experience with numerical systems

caring about, and thinking a lot about, about AI safety

comfortable reading contemporary ML research papers

expertise in security, infrastructure, data, numerics, social science, or one of a dozen other hard-to-find specialities.

This is not a requirements list though. Based on the people working here already, 'great software engineer' and 'easy to get on with' are hard requirements, but the things in the list above are very much nice-to-haves, with several folks having just one or none of them.

Right now our job listings are bucketed into 'security engineer', 'infrastructure engineer', 'research engineer' and the like because these are the noun phrases that a lot of the people we like identify themselves with. But what we're actually most concerned about are generally-great software engineers who - ideally - have some extra bit of deep experience that we lack.

How does engineering compare to research?

At Anthropic there is no hard distinction between researchers and engineers. Some other organisations retain the distinction, but the increasing reliance of research on substantial, custom infrastructure is dissolving the boundary at every industrial lab I'm familiar with.

This might be hard to believe. I think the archetypal research-and-engineering organisation is one where the researchers come up with the fun prototypes, and then toss them over the wall to the engineers to clean up and implement. I think the archetype is common enough that it dissuades a lot of engineers from applying to engineering roles, instead applying to research...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Despite billions of extra funding, small donors can still have a significant impact, published by Benjamin Todd on November 23, 2021 on The Effective Altruism Forum.

I’ve written about how there’s now a lot more funding committed to effective altruism– about $50bn.

It’s natural to think this means small donors can no longer have much impact, and I’ve seen several cases of people saying they’re not sure whether their donations will do any good, because all the opportunities are being taken by large donors.

However, I think this isn’t right: more donations from small donors still have a significant impact. This means raising additional funding is still of value to the community, and I think earning to give and donating to e.g. the Long Term Future Fund, is a highly impactful thing to do – probably more impactful than the vast majority of careers.

I also think the increase in funding means there’s an opportunity to do even more good than earning to give, and that people earning to give currently should seriously consider switching to the kinds of opportunities flagged in my talk at EAG. But that doesn’t mean that small donations have no impact.

Instead:

What matters is not the total amount of available funding, but the current level of cost-effectiveness at the margin. This has likely declined, but is still high.

Small donors should be able to roughly match large donors in terms of cost-effectiveness by ‘funging’ with them.

Small donors can sometimes beat large donors in terms of cost-effectiveness, and I provide a list of some common ways to do this.

At the end, I’ll make some comments on where I think people should donate.

1. What matters is not total funding available but marginal cost-effectiveness

It’s true that as more funding becomes available, all else equal, we should expect more of the best opportunities to be taken, and for cost-effectiveness to decrease.

However, there is a force which limits the size of this effect: how quickly we’re able to discover new opportunities. Because effective altruism is still small and building capacity, it’s not obvious that cost-effectiveness will decline quickly.

While I think the very best opportunities involve taking a more hits based, longterm focused approach than GiveWell, their recommendations serve as a good starting point to examine these dynamics. GiveWell’s top recommendations probably constitute the ‘bar’ for neartermist work. In a recent post, Open Philanthropy’s Global Health and Wellbeing team expect to find many opportunities above this bar, but for marginal dollars to go to GiveWell.

Overall, GiveWell now seems to be targeting a cost-effectiveness of 8x GiveDirectly or higher for most donations, though about 20% funds will go towards opportunities that are 5-8x as cost-effective as GiveDirectly, and so additional donations should be about this cost-effective.

GiveWell is unsure whether the margin will be closer to 5x than 8x. In the same post, Open Philanthropy says “we currently expect GiveWell’s marginal cost-effectiveness to end up around 7-8x GiveDirectly”.

They also say they believe that GiveWell’s margin has been around 10x GiveDirectly in recent years, so if it declines to 7x, that will be a 30% fall – this is only a modest decline and still very high.

To illustrate, they estimate that donating $4,250 to a charity that’s 8x GiveDirectly is as good as saving the life of a child under five.

With a lognormal distribution of cost-effectiveness, there should be many more opportunities at the 5x level than the 10x level, so it should be possible to deploy a lot more funds as the bar lowers. (Even setting aside the possibility of discovering new highly cost-effective interventions.)

In a worst case scenario, billions could be spent on cash transfers at a level of cost-effectiveness similar to or only a little ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Despite billions of extra funding, small donors can still have a significant impact, published by Benjamin Todd on November 23, 2021 on The Effective Altruism Forum.

I’ve written about how there’s now a lot more funding committed to effective altruism– about $50bn.

It’s natural to think this means small donors can no longer have much impact, and I’ve seen several cases of people saying they’re not sure whether their donations will do any good, because all the opportunities are being taken by large donors.

However, I think this isn’t right: more donations from small donors still have a significant impact. This means raising additional funding is still of value to the community, and I think earning to give and donating to e.g. the Long Term Future Fund, is a highly impactful thing to do – probably more impactful than the vast majority of careers.

I also think the increase in funding means there’s an opportunity to do even more good than earning to give, and that people earning to give currently should seriously consider switching to the kinds of opportunities flagged in my talk at EAG. But that doesn’t mean that small donations have no impact.

Instead:

What matters is not the total amount of available funding, but the current level of cost-effectiveness at the margin. This has likely declined, but is still high.

Small donors should be able to roughly match large donors in terms of cost-effectiveness by ‘funging’ with them.

Small donors can sometimes beat large donors in terms of cost-effectiveness, and I provide a list of some common ways to do this.

At the end, I’ll make some comments on where I think people should donate.

1. What matters is not total funding available but marginal cost-effectiveness

It’s true that as more funding becomes available, all else equal, we should expect more of the best opportunities to be taken, and for cost-effectiveness to decrease.

However, there is a force which limits the size of this effect: how quickly we’re able to discover new opportunities. Because effective altruism is still small and building capacity, it’s not obvious that cost-effectiveness will decline quickly.

While I think the very best opportunities involve taking a more hits based, longterm focused approach than GiveWell, their recommendations serve as a good starting point to examine these dynamics. GiveWell’s top recommendations probably constitute the ‘bar’ for neartermist work. In a recent post, Open Philanthropy’s Global Health and Wellbeing team expect to find many opportunities above this bar, but for marginal dollars to go to GiveWell.

Overall, GiveWell now seems to be targeting a cost-effectiveness of 8x GiveDirectly or higher for most donations, though about 20% funds will go towards opportunities that are 5-8x as cost-effective as GiveDirectly, and so additional donations should be about this cost-effective.

GiveWell is unsure whether the margin will be closer to 5x than 8x. In the same post, Open Philanthropy says “we currently expect GiveWell’s marginal cost-effectiveness to end up around 7-8x GiveDirectly”.

They also say they believe that GiveWell’s margin has been around 10x GiveDirectly in recent years, so if it declines to 7x, that will be a 30% fall – this is only a modest decline and still very high.

To illustrate, they estimate that donating $4,250 to a charity that’s 8x GiveDirectly is as good as saving the life of a child under five.

With a lognormal distribution of cost-effectiveness, there should be many more opportunities at the 5x level than the 10x level, so it should be possible to deploy a lot more funds as the bar lowers. (Even setting aside the possibility of discovering new highly cost-effective interventions.)

In a worst case scenario, billions could be spent on cash transfers at a level of cost-effectiveness similar to or only a little ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Reducing EA Jargon & How to Do It, published by Akash on November 22, 2021 on The Effective Altruism Forum.

TLDR: Jargon often worsens the communication of EA ideas and makes it harder for us to update our models of the world. I think EAs should apply strategies to notice/reduce jargon, and I offer a few examples of jargon-reducing techniques.

A few weeks ago, I attended EA Global and a retreat about community building. At both events, we discussed EA-related ideas and how to most effectively communicate them.

One idea I’ve been reflecting on: It can be tempting to use jargon when explaining EA ideas, even in contexts in which jargon is not helpful.

In this post, I describe some drawbacks of EA jargon & offer some suggestions for EAs who want to reduce their use of jargon. (Note: Some of these ideas are inspired by Robert Wilbin's talk and forum post about EA jargon. I have tried to avoid covering the same points he raises, but I strongly suggest checking out those resources.)

What is jargon?

Jargon is “special words or expressions that are used by a particular profession or group and are difficult for others to understand.” It can refer to any terms/phrases that are used in shorthand to communicate broader ideas. Examples include: “Epistemic humility,” “Shapley values,” “Population ethics,” “The INT framework,” and “longtermism.”

Why should we reduce jargon?

Spreading knowledge more efficiently. Reducing jargon will increase the rate at which ideas are spread (especially to people who are not already in the EA community). Logic: Less Jargon → Clearer messages → more people understand them → faster rate of information transfer → more EAs!

Spreading knowledge more accurately (and reducing low-fidelity messaging). Jargon can make it more likely for people to misinterpret EA ideas. Misinterpretations are generally bad, and they can be really bad if influential communicators (e.g., journalists) disseminate misinterpretations of EA concepts to large numbers of people.

These two benefits have focused largely on how others react to jargon. The next two focus on how jargon may directly benefit the person who is communicating:

Improving our own reasoning. In some cases, jargon can improve precision. In others, jargon makes ideas less precise. Consider claims like: “Factory farming is a neglected cause area” or “AI safety is important if you believe in longtermism.” These claims would be clearer—and the speaker would be forced to think more deeply about their position—if we removed the italicized word and replaced it with a more specific and precise belief (see this post for more examples of imprecise/misused jargon).

Making it easier for others to prove us wrong. I have found that others disagree with me more frequently when I deliberately avoid jargon. There are probably two main reasons why.

First, people are often reluctant to admit that they don’t know something. There’s a strong temptation to simply nod along and agree, especially for new members of a group, even if they’re deeply confused.

Second, jargon can make it seem like people agree, even though they actually disagree. This is because jargon is ambiguous and can often be interpreted in many ways. Consider the statement, “Factory farming is a neglected cause area.” The speaker could mean any of the following:

Outside of the EA community, there are not enough people or organizations focused on fighting factory farming.

Within the EA community, there are not many people or organizations focused on fighting factory farming.

There is not enough money devoted to fighting factory farming.

There are not enough people fighting factory farming.

There are not enough individuals with a specific set of skills fighting factory farming.

Notice that two people could agree with the broad claim “factory farming is a ne...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Reducing EA Jargon & How to Do It, published by Akash on November 22, 2021 on The Effective Altruism Forum.

TLDR: Jargon often worsens the communication of EA ideas and makes it harder for us to update our models of the world. I think EAs should apply strategies to notice/reduce jargon, and I offer a few examples of jargon-reducing techniques.

A few weeks ago, I attended EA Global and a retreat about community building. At both events, we discussed EA-related ideas and how to most effectively communicate them.

One idea I’ve been reflecting on: It can be tempting to use jargon when explaining EA ideas, even in contexts in which jargon is not helpful.

In this post, I describe some drawbacks of EA jargon & offer some suggestions for EAs who want to reduce their use of jargon. (Note: Some of these ideas are inspired by Robert Wilbin's talk and forum post about EA jargon. I have tried to avoid covering the same points he raises, but I strongly suggest checking out those resources.)

What is jargon?

Jargon is “special words or expressions that are used by a particular profession or group and are difficult for others to understand.” It can refer to any terms/phrases that are used in shorthand to communicate broader ideas. Examples include: “Epistemic humility,” “Shapley values,” “Population ethics,” “The INT framework,” and “longtermism.”

Why should we reduce jargon?

Spreading knowledge more efficiently. Reducing jargon will increase the rate at which ideas are spread (especially to people who are not already in the EA community). Logic: Less Jargon → Clearer messages → more people understand them → faster rate of information transfer → more EAs!

Spreading knowledge more accurately (and reducing low-fidelity messaging). Jargon can make it more likely for people to misinterpret EA ideas. Misinterpretations are generally bad, and they can be really bad if influential communicators (e.g., journalists) disseminate misinterpretations of EA concepts to large numbers of people.

These two benefits have focused largely on how others react to jargon. The next two focus on how jargon may directly benefit the person who is communicating:

Improving our own reasoning. In some cases, jargon can improve precision. In others, jargon makes ideas less precise. Consider claims like: “Factory farming is a neglected cause area” or “AI safety is important if you believe in longtermism.” These claims would be clearer—and the speaker would be forced to think more deeply about their position—if we removed the italicized word and replaced it with a more specific and precise belief (see this post for more examples of imprecise/misused jargon).

Making it easier for others to prove us wrong. I have found that others disagree with me more frequently when I deliberately avoid jargon. There are probably two main reasons why.

First, people are often reluctant to admit that they don’t know something. There’s a strong temptation to simply nod along and agree, especially for new members of a group, even if they’re deeply confused.

Second, jargon can make it seem like people agree, even though they actually disagree. This is because jargon is ambiguous and can often be interpreted in many ways. Consider the statement, “Factory farming is a neglected cause area.” The speaker could mean any of the following:

Outside of the EA community, there are not enough people or organizations focused on fighting factory farming.

Within the EA community, there are not many people or organizations focused on fighting factory farming.

There is not enough money devoted to fighting factory farming.

There are not enough people fighting factory farming.

There are not enough individuals with a specific set of skills fighting factory farming.

Notice that two people could agree with the broad claim “factory farming is a ne...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Integrating Three Models of (Human) Cognition, published by Jack Koch on November 23, 2021 on The AI Alignment Forum.

You may have heard a few things about “predictive processing” or “the global neuronal workspace,” and you may have read some of Steve Byrnes’ excellent posts about what’s going on computationally in the human brain. But how does it all fit together? How can we begin to arrive at a unified computational picture of human intelligence, and how can it inform our efforts in AI safety and alignment? In this post, I endeavor to take steps towards these ends by integrating insights from three computational frameworks for modeling what’s going on in the human brain, with the hope of laying an adequate foundation for more precisely talking about how the brain cognitively implements goal-directed behaviors:

Predictive Processing: how perception, cognition, and action arise from hierarchical probabilistic generative modeling in the brain

Global Neuronal Workspace: a functional model of access consciousness, i.e. of information that enters our awareness and may subsequently be reported upon

Steve Byrnes’ computational framework: a lot of excellent (in my opinion) work on human cognition, especially motivation, all with an eye toward AI safety. I want to use a better understanding of human cognition to speak more precisely about how things like “goals” and “desires” are implemented, and Steve’s overall framework overlaps quite naturally with the previous two, so I couldn’t leave this one out!

This is a somewhat long post, so I will go ahead and flag the structure here, in case some readers want to skim and/or skip to sections that sound most interesting to them:

Motivations for this project

An overview of each of the three frameworks.

Here, I tried to assume fairly little previous familiarity with the frameworks and attempted to provide a decent foundation in each, while making sure to emphasize the aspects most relevant for the purposes of this post.

Fitting things together from the three frameworks.

How to think about “predictive processing” in the context of Steve’s framework

GNW in the context of the other two frameworks, especially PP, including a better understanding of how it’s actually implemented

Conclusion

This work was completed as Summer Research Fellow at the Center on Long-Term Risk under the mentorship of Richard Ngo. Thanks to Richard, Steve Byrnes, Alex Fabbri, and Euan McLean for feedback on drafts of this post.

Motivation

In thinking about existential risk from AIs, we’re usually fundamentally worried about something like “(capable) AIs have the wrong goals and cause catastrophes as a result.”

I want to better understand how the human mind cognitively implements goal-directed behaviors. I believe this would have several benefits in the context of AI safety and alignment. For one, it facilitates a better understanding of human goals, values, and intentions—in other words, “what we want to align AIs with.” Additionally, a better understanding of goals and agency as implemented in human minds could plausibly help us build more naturally-alignable systems.

The main reason for my interest, though, is that having a better understanding of how biological neural nets implement intelligent goal-directed behaviors will hopefully allow us to, by drawing the right connections to artificial neural nets, further constrain our expectations about how advanced ML models could implement similar behaviors, facilitating the construction of intentional stance models that generalize better out-of-distribution by more closely mirroring the causal dynamics producing agent behavior. We need to have a much better understanding of how (artificial) neural networks will cognitively implement goal-directed behavior if we want to predict how they will perform out-of-distributi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Integrating Three Models of (Human) Cognition, published by Jack Koch on November 23, 2021 on The AI Alignment Forum.

You may have heard a few things about “predictive processing” or “the global neuronal workspace,” and you may have read some of Steve Byrnes’ excellent posts about what’s going on computationally in the human brain. But how does it all fit together? How can we begin to arrive at a unified computational picture of human intelligence, and how can it inform our efforts in AI safety and alignment? In this post, I endeavor to take steps towards these ends by integrating insights from three computational frameworks for modeling what’s going on in the human brain, with the hope of laying an adequate foundation for more precisely talking about how the brain cognitively implements goal-directed behaviors:

Predictive Processing: how perception, cognition, and action arise from hierarchical probabilistic generative modeling in the brain

Global Neuronal Workspace: a functional model of access consciousness, i.e. of information that enters our awareness and may subsequently be reported upon

Steve Byrnes’ computational framework: a lot of excellent (in my opinion) work on human cognition, especially motivation, all with an eye toward AI safety. I want to use a better understanding of human cognition to speak more precisely about how things like “goals” and “desires” are implemented, and Steve’s overall framework overlaps quite naturally with the previous two, so I couldn’t leave this one out!

This is a somewhat long post, so I will go ahead and flag the structure here, in case some readers want to skim and/or skip to sections that sound most interesting to them:

Motivations for this project

An overview of each of the three frameworks.

Here, I tried to assume fairly little previous familiarity with the frameworks and attempted to provide a decent foundation in each, while making sure to emphasize the aspects most relevant for the purposes of this post.

Fitting things together from the three frameworks.

How to think about “predictive processing” in the context of Steve’s framework

GNW in the context of the other two frameworks, especially PP, including a better understanding of how it’s actually implemented

Conclusion

This work was completed as Summer Research Fellow at the Center on Long-Term Risk under the mentorship of Richard Ngo. Thanks to Richard, Steve Byrnes, Alex Fabbri, and Euan McLean for feedback on drafts of this post.

Motivation

In thinking about existential risk from AIs, we’re usually fundamentally worried about something like “(capable) AIs have the wrong goals and cause catastrophes as a result.”

I want to better understand how the human mind cognitively implements goal-directed behaviors. I believe this would have several benefits in the context of AI safety and alignment. For one, it facilitates a better understanding of human goals, values, and intentions—in other words, “what we want to align AIs with.” Additionally, a better understanding of goals and agency as implemented in human minds could plausibly help us build more naturally-alignable systems.

The main reason for my interest, though, is that having a better understanding of how biological neural nets implement intelligent goal-directed behaviors will hopefully allow us to, by drawing the right connections to artificial neural nets, further constrain our expectations about how advanced ML models could implement similar behaviors, facilitating the construction of intentional stance models that generalize better out-of-distribution by more closely mirroring the causal dynamics producing agent behavior. We need to have a much better understanding of how (artificial) neural networks will cognitively implement goal-directed behavior if we want to predict how they will perform out-of-distributi...