Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Important ideas for prioritizing ambitious funding opportunities, published by jh on December 3, 2021 on The Effective Altruism Forum.

At the recent EA Global, Will MacAskill called for the community to be ambitious and aim for non-marginal impacts. Ben Todd of 80,000 hours has also made ambition a theme, especially in regards to career choice. In this post, I develop some key ideas that seem important for prioritizing ambitious funding opportunities.

If you care about the social impact of your funding then you need to care about how other funders react to your actions. Scale, Neglectedness, Solvability (SNS) framing, at least as usually explained, doesn't account for such considerations. This could be an issue now that EA wants to pursue ambitious projects that will overlap with more stakeholders.

To address this issue, I have developed a model for impact in equilibrium. In the past I’ve only discussed this with technical audiences. For this post I've put together diagrams to explain the key ideas.

The main points are:

Scalability. That is, how much the effectiveness of a funding opportunity, whether nonprofit or for profit, persists with scale. Funding GiveDirectly is an example of an opportunity that is regarded as scaling relatively well. Scalability is already an implicit part of Solvability in SNS, but it stands out on its own in general equilibrium.

Urgency versus patience. To determine the ‘urgency’ of giving now, we need to compare it to 'investing to give' to the best funding opportunity in the future. This determines how much you should value impacts now compared to financial returns. In turn, this is crucial to assessing the tradeoffs involved in impact investing. Differences in urgency between funders will determine who funds what and how much.

Risk. Risk is a key driver of for-profit enterprise size. If we allow for 'altruistic' risk too, then risk can also be a key consideration for nonprofit enterprises. Part of the impact of funding an opportunity is helping the market by sharing the risk.

New takes on neglectedness. New ways of looking at neglectedness emerge in equilibrium. 'Investment neglectedness' is a ratio that highlights that it is more impactful to fund a project when its existing funders are less wealthy. But, all else equal (given the same access to funding), it is a positive sign if a project has already been able to attract investment. 'Value neglectedness' is a ratio that highlights that your impact on a funding opportunity will be greater when you value its impacts more than other funders.

This post is long but I feel that is necessary to cover all the main ideas. I plan to cover specific applications of these concepts in shorter posts in the future.

All sorts of questions, comments, and feedback on this post are very welcome and would be appreciated.

Background

This post is based on my work at the Total Portfolio Project. Our goal is to help EAs and aligned investors prioritize the most impactful funding opportunities. Whether these opportunities are grants, 'investing to give' or impact investments. Projects that we've completed so far range from theoretical research (like in this post), to advising on high impact investment deals, to strategic research on balancing 'give now versus later' for new major EA funders.

We see many investment-related research topics that are important, neglected, and tractable. Especially for financial professionals and academics who have a deep enough understanding of EA thinking (and we aim to help build this community). Topics we see as neglected include ‘give now versus later’, patient philanthropy, altruistic leverage and mission correlation.

As a step towards getting EA-aligned ideas considered by leading financial economists, I’ve prepared these two working papers. The goal of this post is to prese...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shulman and Yudkowsky on AI progress, published by CarlShulman on December 4, 2021 on The Effective Altruism Forum.

This post is a transcript of a discussion between Carl Shulman and Eliezer Yudkowsky, following up on a conversation with Paul Christiano and Ajeya Cotra.

Color key:

9.14. Carl Shulman's predictions

[Shulman][20:30]

I'll interject some points re the earlier discussion about how animal data relates to the 'AI scaling to AGI' thesis.

1. In humans it's claimed the IQ-job success correlation varies by job, For a scientist or doctor it might be 0.6+, for a low complexity job more like 0.4, or more like 0.2 for simple repetitive manual labor. That presumably goes down a lot with less in the way of hands, or focused on low density foods like baleen whales or grazers. If it's 0.1 for animals like orcas or elephants, or 0.05, then there's 4-10x less fitness return to smarts.

2. But they outmass humans by more than 4-10x. Elephants 40x, orca 60x+. Metabolically (20 watts divided by BMR of the animal) the gap is somewhat smaller though, because of metabolic scaling laws (energy scales with 3/4 or maybe 2/3 power, so ).

%27s_law

If dinosaurs were poikilotherms, that's a 10x difference in energy budget vs a mammal of the same size, although there is debate about their metabolism.

3. If we're looking for an innovation in birds and primates, there's some evidence of 'hardware' innovation rather than 'software.' Herculano-Houzel reports in The Human Advantage (summarizing much prior work neuron counting) different observational scaling laws for neuron number with brain mass for different animal lineages.

We were particularly interested in cellular scaling differences that might have arisen in primates. If the same rules relating numbers of neurons to brain size in rodents (6)

The brain of the capuchin monkey, for instance, weighing 52 g, contains >3× more neurons in the cerebral cortex and ≈2× more neurons in the cerebellum than the larger brain of the capybara, weighing 76 g.

[Editor’s Note: Quote source is “Cellular scaling rules for primate brains.”]

In rodents brain mass increases with neuron count n^1.6, whereas it's close to linear (n^1.1) in primates. For cortex neurons and cortex mass 1.7 and 1.0. In general birds and primates are outliers in neuron scaling with brain mass.

Note also that bigger brains with lower neuron density have longer communication times from one side of the brain to the other. So primates and birds can have faster clock speeds for integrated thought than a large elephant or whale with similar neuron count.

4. Elephants have brain mass ~2.5x human, and 3x neurons, but 98% of those are in the cerebellum (vs 80% in or less in most animals; these are generally the tiniest neurons and seem to do a bunch of fine motor control). Human cerebral cortex has 3x the neurons of the elephant cortex (which has twice the mass). The giant cerebellum seems like controlling the very complex trunk.

Blue whales get close to human neuron counts with much larger brains.

5. As Paul mentioned, human brain volume correlation with measures of cognitive function after correcting for measurement error on the cognitive side is in the vicinity of 0.3-0.4 (might go a bit higher after controlling for non-functional brain volume variation, lower from removing confounds). The genetic correlation with cognitive function in this study is 0.24:

So it accounts for a minority of genetic influences on cognitive ability. We'd also expect a bunch of genetic variance that's basically disruptive mutations in mutation-selection balance (e.g. schizophrenia seems to be a result of that, with schizophrenia alleles under negative selection, but a big mutational target, with the standing burden set by the level of fitness penalty for it; in niches with less return to cognition the mutatio...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shulman and Yudkowsky on AI progress, published by CarlShulman on December 4, 2021 on The Effective Altruism Forum.

This post is a transcript of a discussion between Carl Shulman and Eliezer Yudkowsky, following up on a conversation with Paul Christiano and Ajeya Cotra.

Color key:

9.14. Carl Shulman's predictions

[Shulman][20:30]

I'll interject some points re the earlier discussion about how animal data relates to the 'AI scaling to AGI' thesis.

1. In humans it's claimed the IQ-job success correlation varies by job, For a scientist or doctor it might be 0.6+, for a low complexity job more like 0.4, or more like 0.2 for simple repetitive manual labor. That presumably goes down a lot with less in the way of hands, or focused on low density foods like baleen whales or grazers. If it's 0.1 for animals like orcas or elephants, or 0.05, then there's 4-10x less fitness return to smarts.

2. But they outmass humans by more than 4-10x. Elephants 40x, orca 60x+. Metabolically (20 watts divided by BMR of the animal) the gap is somewhat smaller though, because of metabolic scaling laws (energy scales with 3/4 or maybe 2/3 power, so ).

%27s_law

If dinosaurs were poikilotherms, that's a 10x difference in energy budget vs a mammal of the same size, although there is debate about their metabolism.

3. If we're looking for an innovation in birds and primates, there's some evidence of 'hardware' innovation rather than 'software.' Herculano-Houzel reports in The Human Advantage (summarizing much prior work neuron counting) different observational scaling laws for neuron number with brain mass for different animal lineages.

We were particularly interested in cellular scaling differences that might have arisen in primates. If the same rules relating numbers of neurons to brain size in rodents (6)

The brain of the capuchin monkey, for instance, weighing 52 g, contains >3× more neurons in the cerebral cortex and ≈2× more neurons in the cerebellum than the larger brain of the capybara, weighing 76 g.

[Editor’s Note: Quote source is “Cellular scaling rules for primate brains.”]

In rodents brain mass increases with neuron count n^1.6, whereas it's close to linear (n^1.1) in primates. For cortex neurons and cortex mass 1.7 and 1.0. In general birds and primates are outliers in neuron scaling with brain mass.

Note also that bigger brains with lower neuron density have longer communication times from one side of the brain to the other. So primates and birds can have faster clock speeds for integrated thought than a large elephant or whale with similar neuron count.

4. Elephants have brain mass ~2.5x human, and 3x neurons, but 98% of those are in the cerebellum (vs 80% in or less in most animals; these are generally the tiniest neurons and seem to do a bunch of fine motor control). Human cerebral cortex has 3x the neurons of the elephant cortex (which has twice the mass). The giant cerebellum seems like controlling the very complex trunk.

Blue whales get close to human neuron counts with much larger brains.

5. As Paul mentioned, human brain volume correlation with measures of cognitive function after correcting for measurement error on the cognitive side is in the vicinity of 0.3-0.4 (might go a bit higher after controlling for non-functional brain volume variation, lower from removing confounds). The genetic correlation with cognitive function in this study is 0.24:

So it accounts for a minority of genetic influences on cognitive ability. We'd also expect a bunch of genetic variance that's basically disruptive mutations in mutation-selection balance (e.g. schizophrenia seems to be a result of that, with schizophrenia alleles under negative selection, but a big mutational target, with the standing burden set by the level of fitness penalty for it; in niches with less return to cognition the mutatio...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Privacy and Manipulation, published by Raemon on December 5, 2021 on LessWrong.

Previously:

“Can you keep that confidential? How do you know?”

Parameters of Privacy

Norm Innovation and Theory of Mind

My parents taught me the norm of keeping my promises.

My vague societal culture taught me a norm of automatically treat certain types of information as private.

My vague rationalist culture taught me norms that include:

noticing when I'm confused

noticing when societal norms were inadequate to handle my situation

being very honest, and having strong theoretical underpinnings for handling situations where I felt it was inadvisable to be honest.

Eliezer's post about meta-honesty was one of the most influential posts I've read in the past few years, and among the posts that inspired the coordination frontier. I was impressed that Eliezer looked at ethical edgecases, and wasn't content to make a reasonable judgment call and declare himself done.

He went on to think through the ramifications of various policies, devise a potential new norm/protocol, examine reasons that protocol might work or not work. He noted considerations like [paraphrased] "It matters that the norm be simple enough that people can reliably understand and use it." Or, quoted directly: "This norm is too subtle for Twitter. It might be too subtle for us, too."

From this post, I derived a (not-quite-spelled-out) norm of "when you try to navigate the edge cases of your norms, try thinking through the underlying principles. But don't try to be too clever, and consider the ways your edge-case-handling may fail to scale."

With this in mind, I want to walk through one of the ethical dilemmas I faced that I reflected on, when writing Norm Innovation and Theory of Mind. This is more of an object-level post, primarily a followup to my Privacy Practices sequence. But it seemed like a useful illustrative example for the Coordination Frontier concept.

Privacy norms can be wielded as an obfuscating weapon

Sometimes, privacy is wielded as a tool to enable manipulation.

I’ve run into a couple people who exploited my good faith / willingness to keep things confidential, as part of an overall manipulative pattern. Unfortunately, I don't feel comfortable going too far into the details here (please don't speculate in the comments), which makes it a bit harder to sanity check.

"Manipulation" is a tricky to define. It's a blurry line between "communicating normally" and "communicating in a way that systematically distorts another people's thinking and controls their behavior against their wishes". I'd like it say "it's hard to define but you know it when you see it", but often it's hard to see it because manipulation systematically tries not to be seen.

I've met some people seemed deliberately manipulative, and some people who might have been well intentioned, but in the end it didn’t matter. They interacted with me (and others) in a way that felt increasingly uncomfortable, which seemed to be harming people. They skirted lines, wove narratives that made it feel awkward for me to criticize them or think clearly.

One of the key strategies they employed was to make it feel awkward to get help from other people to think or sanity check things. And one tactic in that strategy was pushing for confidentiality – sometimes explicitly, sometimes implicitly.

Explicit promises I regret

One person (call them Dave) asked for explicit promises of confidentiality on a number of occasions, sometimes after-the-fact. We once had a long conversation about their worldview and worries they had, which ended with me saying "so, wait, is this meant to be confidential?". They responded, somewhat alarmed-seeming, with "oh definitely I would never have told you all this if I thought you might share it."

At the time I found that somewhat annoying, but agr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Privacy and Manipulation, published by Raemon on December 5, 2021 on LessWrong.

Previously:

“Can you keep that confidential? How do you know?”

Parameters of Privacy

Norm Innovation and Theory of Mind

My parents taught me the norm of keeping my promises.

My vague societal culture taught me a norm of automatically treat certain types of information as private.

My vague rationalist culture taught me norms that include:

noticing when I'm confused

noticing when societal norms were inadequate to handle my situation

being very honest, and having strong theoretical underpinnings for handling situations where I felt it was inadvisable to be honest.

Eliezer's post about meta-honesty was one of the most influential posts I've read in the past few years, and among the posts that inspired the coordination frontier. I was impressed that Eliezer looked at ethical edgecases, and wasn't content to make a reasonable judgment call and declare himself done.

He went on to think through the ramifications of various policies, devise a potential new norm/protocol, examine reasons that protocol might work or not work. He noted considerations like [paraphrased] "It matters that the norm be simple enough that people can reliably understand and use it." Or, quoted directly: "This norm is too subtle for Twitter. It might be too subtle for us, too."

From this post, I derived a (not-quite-spelled-out) norm of "when you try to navigate the edge cases of your norms, try thinking through the underlying principles. But don't try to be too clever, and consider the ways your edge-case-handling may fail to scale."

With this in mind, I want to walk through one of the ethical dilemmas I faced that I reflected on, when writing Norm Innovation and Theory of Mind. This is more of an object-level post, primarily a followup to my Privacy Practices sequence. But it seemed like a useful illustrative example for the Coordination Frontier concept.

Privacy norms can be wielded as an obfuscating weapon

Sometimes, privacy is wielded as a tool to enable manipulation.

I’ve run into a couple people who exploited my good faith / willingness to keep things confidential, as part of an overall manipulative pattern. Unfortunately, I don't feel comfortable going too far into the details here (please don't speculate in the comments), which makes it a bit harder to sanity check.

"Manipulation" is a tricky to define. It's a blurry line between "communicating normally" and "communicating in a way that systematically distorts another people's thinking and controls their behavior against their wishes". I'd like it say "it's hard to define but you know it when you see it", but often it's hard to see it because manipulation systematically tries not to be seen.

I've met some people seemed deliberately manipulative, and some people who might have been well intentioned, but in the end it didn’t matter. They interacted with me (and others) in a way that felt increasingly uncomfortable, which seemed to be harming people. They skirted lines, wove narratives that made it feel awkward for me to criticize them or think clearly.

One of the key strategies they employed was to make it feel awkward to get help from other people to think or sanity check things. And one tactic in that strategy was pushing for confidentiality – sometimes explicitly, sometimes implicitly.

Explicit promises I regret

One person (call them Dave) asked for explicit promises of confidentiality on a number of occasions, sometimes after-the-fact. We once had a long conversation about their worldview and worries they had, which ended with me saying "so, wait, is this meant to be confidential?". They responded, somewhat alarmed-seeming, with "oh definitely I would never have told you all this if I thought you might share it."

At the time I found that somewhat annoying, but agr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Crosspost] Reducing Risks of Astronomical Suffering: A Neglected Priority, published by Bob Jacobs on December 4, 2021 on The Effective Altruism Forum.

This is a review crosspost for the Forum Review. It was first published at 14 September 2016 by David Althaus and Lukas Gloor

Summary

Will we go extinct, or will we succeed in building a flourishing utopia? Discussions about the future trajectory of humanity often center around these two possibilities, which tends to ignore that survival does not always imply utopian outcomes, or that outcomes where humans go extinct could differ tremendously in how much suffering they contain. One major risk is that space colonization, either through humans or (misaligned) artificial intelligence, may end up producing cosmically significant amounts of suffering. For a variety of reasons, such scenarios are rarely discussed and often underestimated. The neglectedness of risks of cosmically significant amounts of suffering (“suffering risks” or “s-risks”) makes their reduction a plausible priority from the perspective of many value systems. Rather than focusing exclusively on ensuring that there will be a future, we recommend interventions that improve the future’s overall quality.

I. Introduction

Among actors and organizations concerned with shaping the “long-term future,” the discourse has so far been centered around the concept of existential risks. “Existential risk” (or “x-risk”) was defined by Bostrom (2002) as “[...] an adverse outcome [which] would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.”

This definition is unfortunate in that it lumps together events that lead to vast amounts of suffering and events that lead to the extinction (or failure to reach the potential) of humanity. However, many value systems would agree that extinction is not the worst possible outcome, and that avoiding large quantities of suffering is of utmost moral importance.

We should differentiate between existential risks (i.e., risks of “mere” extinction or failed potential) and suffering risks.1. Suffering risks are risks of events that bring about suffering in cosmically significant amounts. By “significant”, we mean significant relative to expected future suffering. Note that it may turn out that the amount of suffering that we can influence is dwarfed by suffering that we can’t influence. By “expected future suffering” we mean “expected action-relevant suffering in the future”.

The above distinctions are all the more important because the term “existential risk” has often been used interchangeably with “risks of extinction”, omitting any reference to the future’s quality.2 Finally, some futures may contain both vast amounts of happiness and vast amounts of suffering, which constitutes an s-risk but not necessarily a (severe) x-risk. For instance, an event that would create 1025 unhappy beings in a future that already contains 1035 happy individuals constitutes an s-risk, but not an x-risk.3

The Case for Suffering-Focused Ethics outlined several reasons for considering suffering reduction one’s primary moral priority. From this perspective in particular, s-risks should be addressed before addressing extinction risks. Reducing extinction risks makes it more likely that there will be a future, possibly one involving space colonization and the astronomical stakes that come with it. But it often does not robustly improve the quality of the future, i.e., how much suffering or happiness it will likely contain.4 A future with space colonization may contain vastly more sentient minds than have existed so far. If something goes wrong, or even if things do not go “right enough”, this would multiply the total amount of suffering (in our part of the universe) by a huge factor.

Depending on our norma...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Crosspost] Reducing Risks of Astronomical Suffering: A Neglected Priority, published by Bob Jacobs on December 4, 2021 on The Effective Altruism Forum.

This is a review crosspost for the Forum Review. It was first published at 14 September 2016 by David Althaus and Lukas Gloor

Summary

Will we go extinct, or will we succeed in building a flourishing utopia? Discussions about the future trajectory of humanity often center around these two possibilities, which tends to ignore that survival does not always imply utopian outcomes, or that outcomes where humans go extinct could differ tremendously in how much suffering they contain. One major risk is that space colonization, either through humans or (misaligned) artificial intelligence, may end up producing cosmically significant amounts of suffering. For a variety of reasons, such scenarios are rarely discussed and often underestimated. The neglectedness of risks of cosmically significant amounts of suffering (“suffering risks” or “s-risks”) makes their reduction a plausible priority from the perspective of many value systems. Rather than focusing exclusively on ensuring that there will be a future, we recommend interventions that improve the future’s overall quality.

I. Introduction

Among actors and organizations concerned with shaping the “long-term future,” the discourse has so far been centered around the concept of existential risks. “Existential risk” (or “x-risk”) was defined by Bostrom (2002) as “[...] an adverse outcome [which] would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.”

This definition is unfortunate in that it lumps together events that lead to vast amounts of suffering and events that lead to the extinction (or failure to reach the potential) of humanity. However, many value systems would agree that extinction is not the worst possible outcome, and that avoiding large quantities of suffering is of utmost moral importance.

We should differentiate between existential risks (i.e., risks of “mere” extinction or failed potential) and suffering risks.1. Suffering risks are risks of events that bring about suffering in cosmically significant amounts. By “significant”, we mean significant relative to expected future suffering. Note that it may turn out that the amount of suffering that we can influence is dwarfed by suffering that we can’t influence. By “expected future suffering” we mean “expected action-relevant suffering in the future”.

The above distinctions are all the more important because the term “existential risk” has often been used interchangeably with “risks of extinction”, omitting any reference to the future’s quality.2 Finally, some futures may contain both vast amounts of happiness and vast amounts of suffering, which constitutes an s-risk but not necessarily a (severe) x-risk. For instance, an event that would create 1025 unhappy beings in a future that already contains 1035 happy individuals constitutes an s-risk, but not an x-risk.3

The Case for Suffering-Focused Ethics outlined several reasons for considering suffering reduction one’s primary moral priority. From this perspective in particular, s-risks should be addressed before addressing extinction risks. Reducing extinction risks makes it more likely that there will be a future, possibly one involving space colonization and the astronomical stakes that come with it. But it often does not robustly improve the quality of the future, i.e., how much suffering or happiness it will likely contain.4 A future with space colonization may contain vastly more sentient minds than have existed so far. If something goes wrong, or even if things do not go “right enough”, this would multiply the total amount of suffering (in our part of the universe) by a huge factor.

Depending on our norma...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misc. questions about EfficientZero, published by Daniel Kokotajlo on December 4, 2021 on LessWrong.

Perhaps these can be thought of as homework questions -- when I imagine us successfully making AI go well, I imagine us building expertise such that we can answer these questions quickly and easily. Before I read the answers I'm going to think for 10min or so about each one and post my own guesses.

Useful links / background reading: The glorious EfficientZero: How it Works. Related comment. EfficientZero GitHub. LW discussion.

Some of these questions are about EfficientZero, the net trained recently; others are about EfficientZero the architecture, imagined to be suitably scaled up to AGI levels. "If we made a much bigger and longer-trained version of this (with suitable training environment) such that it was superhuman AGI..."

EfficientZero vs. reward hacking and inner alignment failure:

Barring inner alignment failure, it’ll eventually reward hack, right? That is, if it gets sufficiently knowledgeable and capable, it’ll realize that it can get loads of reward by hacking its reward channel, and then its core algorithm would evaluate that action/plan highly and do it. Right?

But fortunately (?) maybe there would be an inner alignment failure and the part of it that predicts reward would predict low reward from that action, even in the limit of knowledge and capability? Because it’s learned to predict proxies for reward rather than reward itself, and continued to do so even as it got smarter and more capable? (Would this happen? Why would this not be corrected by further training? Has some sort of proxy crystallization set in? How?)

EfficientZero approximates evidential decision theory, right?

EfficientZero is a consequentialist (in the sense defined here) architecture, right? It’s not, for example, updateless or deontological. For example, it has no deontological constraints except by accident (i.e. if its predictor-net mistakenly predicted super low reward for actions of type X, always, even in cases where actually a reasonable intelligent predictor would predict high reward.) Right?

What is the most complex environment AIs in the family of MuZero, EfficientZero, etc. have been trained on? Is it just some Atari game?

Roughly how many parameters does EfficientZero have? If you don’t know, what about MuZero? What about the biggest net to date from that general family? The EfficientZero paper doesn't give a direct answer but it describes the architecture in enough detail that you might be able to calculate it...

If we kept scaling up EfficientZero by OOMs in every way, what would happen? Would it eventually get to agenty AGI? / APS-AI? After all, it seems pretty sample-efficient already. What if its sample was an entire lifetime of diverse experiences?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misc. questions about EfficientZero, published by Daniel Kokotajlo on December 4, 2021 on LessWrong.

Perhaps these can be thought of as homework questions -- when I imagine us successfully making AI go well, I imagine us building expertise such that we can answer these questions quickly and easily. Before I read the answers I'm going to think for 10min or so about each one and post my own guesses.

Useful links / background reading: The glorious EfficientZero: How it Works. Related comment. EfficientZero GitHub. LW discussion.

Some of these questions are about EfficientZero, the net trained recently; others are about EfficientZero the architecture, imagined to be suitably scaled up to AGI levels. "If we made a much bigger and longer-trained version of this (with suitable training environment) such that it was superhuman AGI..."

EfficientZero vs. reward hacking and inner alignment failure:

Barring inner alignment failure, it’ll eventually reward hack, right? That is, if it gets sufficiently knowledgeable and capable, it’ll realize that it can get loads of reward by hacking its reward channel, and then its core algorithm would evaluate that action/plan highly and do it. Right?

But fortunately (?) maybe there would be an inner alignment failure and the part of it that predicts reward would predict low reward from that action, even in the limit of knowledge and capability? Because it’s learned to predict proxies for reward rather than reward itself, and continued to do so even as it got smarter and more capable? (Would this happen? Why would this not be corrected by further training? Has some sort of proxy crystallization set in? How?)

EfficientZero approximates evidential decision theory, right?

EfficientZero is a consequentialist (in the sense defined here) architecture, right? It’s not, for example, updateless or deontological. For example, it has no deontological constraints except by accident (i.e. if its predictor-net mistakenly predicted super low reward for actions of type X, always, even in cases where actually a reasonable intelligent predictor would predict high reward.) Right?

What is the most complex environment AIs in the family of MuZero, EfficientZero, etc. have been trained on? Is it just some Atari game?

Roughly how many parameters does EfficientZero have? If you don’t know, what about MuZero? What about the biggest net to date from that general family? The EfficientZero paper doesn't give a direct answer but it describes the architecture in enough detail that you might be able to calculate it...

If we kept scaling up EfficientZero by OOMs in every way, what would happen? Would it eventually get to agenty AGI? / APS-AI? After all, it seems pretty sample-efficient already. What if its sample was an entire lifetime of diverse experiences?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Agents as P₂B Chain Reactions, published by Daniel Kokotajlo on December 4, 2021 on The AI Alignment Forum.

tl;dr: Sometimes planners successfully P2B, kicking off a self-sustaining chain reaction / feedback loop of better and better plans (made possible by better and better world-models, more and more resources, etc.) Whereas fire takes a concentration of heat (a spark) as input and produces a greater concentration of heat as output, agents take a concentration of convergent instrumental resources (e.g. data, money, power) as input and produces a greater concentration as output.

Previously we described the Convergent-Instrumental-Goal-to-rule-them-all, P2B: Plan to P2B Better. The most common way to P2B is to plan to plan again, in the immediate future, but with some new relevant piece of data. For example, suppose I am walking somewhere. I look up from my phone to glance at the door. Why? Because without data about the location of the doorknob, I can’t guide my hands to open the door. This sort of thing doesn’t just happen on short timescales, though; humans (and all realistic agents, I’d wager) are hierarchical planners and medium and long-term plans usually involve acquiring new relevant data as well.

This image just talks about collecting good new data, but there are other important convergent instrumental goals too, of course. Such as staying alive, propagating your goals to other agents, and acquiring money. One could modify the diagram to include these other feedback loops as well — more data and money and power being recruited by sensor-learner-planners to acquire more data and money and power. It’s not that all of these metrics will go up with every cycle around the loop; it’s that some goals/resources are “instrumentally convergent” in that they tend to show up frequently in most real-world P2B loops.

Agents, I say, are P2B chain reactions / P2B feedback loops. They are what happens when planners successfully P2B. Whereas fire is a chain reaction that inputs heat + fuel + oxygen and outputs more heat (and often more oxygen and fuel too, as it expands to envelop more such) agents are chain reactions that input sensor-learner-planners with some amount of data, knowledge, power, money, etc. and output more, better sensor-learner-planners with greater amounts of data, knowledge, power, money, etc.

(I don’t I think this definition fully captures our intuitive concept of agency. Rather, P2B chain reactions seem like a big deal, an important concept worth talking about, and close enough to our intuitive concept of agency that I’m going to appropriate the term until someone pushes back.)

Already you might be able to guess why I think agents are powerful:

Consider how fire is a much bigger deal than baking-soda-vinegar. General/robust feedback loops are way way better/more important than narrow/brittle ones, and sensor-learner-planner makes for a P2B loop that is pretty damn general.

Ok, so we are talking about a kind of chain reaction that takes concentrations of knowledge, power, money, etc. and makes them bigger? That sure sounds like it’ll be relevant to discussions of who’s going to win important competitions over market share, political power, and control of the future!

The next post in this sequence will address the following questions, and more:

OK, so does any feedback loop of accumulating convergent instrumental resources count as an agent then? Presumably not, presumably it has to be resulting from some sort of planning to count as a P2B loop. Earlier you said planning is a family of algorithms. Say more about where the borders of this concept are please!

The post after that will answer the Big Questions:

Why is agency powerful? Why should we expect agent AGIs to outcompete human+tool combos for control of the future? Etc.

Thanks for listening. To help u...