Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paxlovid Remains Illegal: 11/24 Update, published by Zvi on November 24, 2021 on LessWrong.

What Alex Tabarrok called The Paxlovid Paradox is getting noticed by the people who notice such things, and completely ignored by everyone else. I’ve split off this week’s Paxlovid update to help make the situation easier to notice, and easier to remember and reference later, lest we forget.

Supply and Demand

The good news is that we are confident Paxlovid is safe and effective and the purchasing department is acting accordingly. Once we decide it is legal, we’ve secured our supply. The Biden administration has agreed to pay $5 billion for 10 million treatments. I previously thought this was 10 million pills as did the source, but I’ve been informed it was 10 million treatments, which means I was about to be happy to pay ten times what we’re actually paying. Kind of neat.

They’re going to make a profit from saving people’s lives. Outrageous!

The bad news, of course, is that Paxlovid Remains Illegal.

The Paxlovid Paradox

Scott Alexander is the latest to point out that this is murderous madness1. As I observed, and was later quoted at Marginal Revolution:

The trial was stopped due to ‘ethical considerations’ for being too effective. You see, we live in a world in which:

It is illegal to give this drug to any patients, because it hasn’t been proven safe and effective.

It is illegal to continue a trial to study the drug, because it has been proven so safe and effective that it isn’t ethical to not give the drug to half the patients.

Who, if they weren’t in the study, couldn’t get the drug at all, because it is illegal due to not being proven safe and effective yet.

So now no one gets added to the trial so those who would have been definitely don’t get Paxlovid, and are several times more likely to die.

But our treatment of them is now ‘ethical.’

For the rest of time we will now hear about how it was only seven deaths and we can’t be sure Paxlovid works or how well it works, and I expect to spend hours arguing over exactly how much it works.

For the rest of time people will argue the study wasn’t big enough so we don’t know the Paxlovid is safe.

Those arguments will then be used both by people arguing to not take Paxlovid, and people who want to require other interventions because of these concerns.

FDA Delenda Est.

Scott Alexander notes that the current prediction market median approval date is January 1, 2022, with a 92% chance of approval by March. I’ve asked Polymarket to put up a market I would trust somewhat more, which should be up in a few days. My guess is that this the current median estimate is somewhat too pessimistic.

He goes through and quickly destroys many plausible reasons why this delay would make sense. In the comments several attempts were made to raise additional objections, with concerns about the ‘manufacturing process’ being the closest to kind of maybe being able to pretend to be a real consideration (as opposed to the others, which were at best pretending to pretend). I still don’t think it does an especially good job of pretending to be a big enough concern to matter, and was disappointed by Scott’s respect for the objection, which I’ll discuss more in detail below. Whatever pills we have now were somehow manufactured, without expected delays we would have made a lot more of them a lot faster, and we could have dealt with that issue while doing the trial if it was the true bottleneck and the incentives had been reasonable.

This is noteworthy, not because I think it’s identifying the causal mechanism (I think it mostly isn’t) but because it describes a world that makes more sense and stopping the trial unethical but a reasonable response to incentives. If only this were about people who responded so well to pure dollar incentives our problems would be so m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paxlovid Remains Illegal: 11/24 Update, published by Zvi on November 24, 2021 on LessWrong.

What Alex Tabarrok called The Paxlovid Paradox is getting noticed by the people who notice such things, and completely ignored by everyone else. I’ve split off this week’s Paxlovid update to help make the situation easier to notice, and easier to remember and reference later, lest we forget.

Supply and Demand

The good news is that we are confident Paxlovid is safe and effective and the purchasing department is acting accordingly. Once we decide it is legal, we’ve secured our supply. The Biden administration has agreed to pay $5 billion for 10 million treatments. I previously thought this was 10 million pills as did the source, but I’ve been informed it was 10 million treatments, which means I was about to be happy to pay ten times what we’re actually paying. Kind of neat.

They’re going to make a profit from saving people’s lives. Outrageous!

The bad news, of course, is that Paxlovid Remains Illegal.

The Paxlovid Paradox

Scott Alexander is the latest to point out that this is murderous madness1. As I observed, and was later quoted at Marginal Revolution:

The trial was stopped due to ‘ethical considerations’ for being too effective. You see, we live in a world in which:

It is illegal to give this drug to any patients, because it hasn’t been proven safe and effective.

It is illegal to continue a trial to study the drug, because it has been proven so safe and effective that it isn’t ethical to not give the drug to half the patients.

Who, if they weren’t in the study, couldn’t get the drug at all, because it is illegal due to not being proven safe and effective yet.

So now no one gets added to the trial so those who would have been definitely don’t get Paxlovid, and are several times more likely to die.

But our treatment of them is now ‘ethical.’

For the rest of time we will now hear about how it was only seven deaths and we can’t be sure Paxlovid works or how well it works, and I expect to spend hours arguing over exactly how much it works.

For the rest of time people will argue the study wasn’t big enough so we don’t know the Paxlovid is safe.

Those arguments will then be used both by people arguing to not take Paxlovid, and people who want to require other interventions because of these concerns.

FDA Delenda Est.

Scott Alexander notes that the current prediction market median approval date is January 1, 2022, with a 92% chance of approval by March. I’ve asked Polymarket to put up a market I would trust somewhat more, which should be up in a few days. My guess is that this the current median estimate is somewhat too pessimistic.

He goes through and quickly destroys many plausible reasons why this delay would make sense. In the comments several attempts were made to raise additional objections, with concerns about the ‘manufacturing process’ being the closest to kind of maybe being able to pretend to be a real consideration (as opposed to the others, which were at best pretending to pretend). I still don’t think it does an especially good job of pretending to be a big enough concern to matter, and was disappointed by Scott’s respect for the objection, which I’ll discuss more in detail below. Whatever pills we have now were somehow manufactured, without expected delays we would have made a lot more of them a lot faster, and we could have dealt with that issue while doing the trial if it was the true bottleneck and the incentives had been reasonable.

This is noteworthy, not because I think it’s identifying the causal mechanism (I think it mostly isn’t) but because it describes a world that makes more sense and stopping the trial unethical but a reasonable response to incentives. If only this were about people who responded so well to pure dollar incentives our problems would be so m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don’t wait – there’s plenty more need and opportunity today, published by GiveDirectly on November 24, 2021 on The Effective Altruism Forum.

Yesterday, leading charity evaluator GiveWell announced they’re committing more money than ever to effective causes, which we commend. This generosity reflects great news: the world is wealthier than ever before. However, they also decided not to distribute $110M this year because they don’t believe they can find good enough ways to use it now. They will distribute $450M to programs they identified could spend funding over the next few years, but decided any additional donations were best saved for later. Their estimate is that they will roll over a similar amount of money next year. In the midst of a global pandemic that pushed 150 million people into extreme poverty while billionaires’ wealth grew by $5.5 trillion, it’s odd timing.

GiveWell’s rationale was fairly straightforward:

“We plan to roll over about $110M (~20% of our forecasted funds raised) into 2022 because we expect the opportunities this funding would be spent on now are much less cost-effective than those we expect to find over the next few years.” (GiveWell)

To be clear, we didn’t expect GiveWell to direct funds to GiveDirectly this year, and that’s not why we’re chiming in. While GiveWell has considered GiveDirectly a “Top Charity” for nearly a decade, they haven’t allocated funds to people in poverty through GiveDirectly since 2015. We’ve always appreciated GiveWell’s commitment to evidence and rigor, and many of us joined GiveDirectly after first hearing of it from GiveWell. We were motivated to respond precisely because we know first hand that GiveWell’s voice matters: they’ve established themselves as the gold standard charity recommender to answer questions for donors everywhere, like How To Make A Good Charitable Choice On Giving Tuesday.

As we share below, we think GiveWell is thinking too small, undervaluing what can be achieved today, underestimating the costs of waiting, overestimating how much better they’ll allocate funds in the future, and not accounting for the perspectives of people living in poverty. In short, we think the world has an unprecedented opportunity to eliminate most of extreme poverty in our lifetimes, and we worry GiveWell’s decision hinders that effort. Their choice to wait for better opportunities focuses on maximizing what they perceive as the direct impact of their money alone at the cost of conveying a tragically discouraging message about the potential impact of everyone else’s.

There is still immense need and opportunity for impact now

“In 2021, we expect to identify $400M in 8x [as cost-effective] or better opportunities. If our fundraising projections hold, we may have $160M (or more) that we’re unable to spend at our current bar. Given that, we expect to: Direct about $50M to 5-8x funding opportunities [and] Roll over about $110M to grant to future opportunities we expect will be 5x or better.” (GiveWell)

We think GiveWell is missing both the magnitude of extreme poverty and the abundance of the resources we have to fight it. Straining to find $500M or even $1B in worthwhile giving opportunities just doesn’t fit with a world where 700+ million people live in extreme poverty, U.S. charitable giving alone is nearly $500B a year, global official development assistance is another $150B a year, and an additional $500B has been pledged via The Giving Pledge. Even GiveWell’s largest few donors alone have enough wealth that it would take decades to distribute it at the pace GiveWell plans to achieve.

Once GiveWell’s top charitable opportunities are funded (like this year), what is everyone else to do? We’d say: keep giving as effectively as you can. The Life You Can Save recommends 16 organizations working on urgent matters of gl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don’t wait – there’s plenty more need and opportunity today, published by GiveDirectly on November 24, 2021 on The Effective Altruism Forum.

Yesterday, leading charity evaluator GiveWell announced they’re committing more money than ever to effective causes, which we commend. This generosity reflects great news: the world is wealthier than ever before. However, they also decided not to distribute $110M this year because they don’t believe they can find good enough ways to use it now. They will distribute $450M to programs they identified could spend funding over the next few years, but decided any additional donations were best saved for later. Their estimate is that they will roll over a similar amount of money next year. In the midst of a global pandemic that pushed 150 million people into extreme poverty while billionaires’ wealth grew by $5.5 trillion, it’s odd timing.

GiveWell’s rationale was fairly straightforward:

“We plan to roll over about $110M (~20% of our forecasted funds raised) into 2022 because we expect the opportunities this funding would be spent on now are much less cost-effective than those we expect to find over the next few years.” (GiveWell)

To be clear, we didn’t expect GiveWell to direct funds to GiveDirectly this year, and that’s not why we’re chiming in. While GiveWell has considered GiveDirectly a “Top Charity” for nearly a decade, they haven’t allocated funds to people in poverty through GiveDirectly since 2015. We’ve always appreciated GiveWell’s commitment to evidence and rigor, and many of us joined GiveDirectly after first hearing of it from GiveWell. We were motivated to respond precisely because we know first hand that GiveWell’s voice matters: they’ve established themselves as the gold standard charity recommender to answer questions for donors everywhere, like How To Make A Good Charitable Choice On Giving Tuesday.

As we share below, we think GiveWell is thinking too small, undervaluing what can be achieved today, underestimating the costs of waiting, overestimating how much better they’ll allocate funds in the future, and not accounting for the perspectives of people living in poverty. In short, we think the world has an unprecedented opportunity to eliminate most of extreme poverty in our lifetimes, and we worry GiveWell’s decision hinders that effort. Their choice to wait for better opportunities focuses on maximizing what they perceive as the direct impact of their money alone at the cost of conveying a tragically discouraging message about the potential impact of everyone else’s.

There is still immense need and opportunity for impact now

“In 2021, we expect to identify $400M in 8x [as cost-effective] or better opportunities. If our fundraising projections hold, we may have $160M (or more) that we’re unable to spend at our current bar. Given that, we expect to: Direct about $50M to 5-8x funding opportunities [and] Roll over about $110M to grant to future opportunities we expect will be 5x or better.” (GiveWell)

We think GiveWell is missing both the magnitude of extreme poverty and the abundance of the resources we have to fight it. Straining to find $500M or even $1B in worthwhile giving opportunities just doesn’t fit with a world where 700+ million people live in extreme poverty, U.S. charitable giving alone is nearly $500B a year, global official development assistance is another $150B a year, and an additional $500B has been pledged via The Giving Pledge. Even GiveWell’s largest few donors alone have enough wealth that it would take decades to distribute it at the pace GiveWell plans to achieve.

Once GiveWell’s top charitable opportunities are funded (like this year), what is everyone else to do? We’d say: keep giving as effectively as you can. The Life You Can Save recommends 16 organizations working on urgent matters of gl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Laplace's rule of succession, published by Ege Erdil on November 23, 2021 on LessWrong.

Laplace's rule of succession is a commonly used heuristic in forecasting the occurence of events which have little to no historical precedents. The basic problem setup is this: suppose you flip a coin with an unknown bias N=H+T times and get H heads and T tails. We want to infer something about the bias p of the coin from this information, where we define p to be the probability of the coin coming up heads.

One obvious idea is to estimate p by ^p=H/(H+T). This works well if we have some decent number of heads and tails, but it fails badly in the edge case where the tosses are very imbalanced. In the extreme case, if we get all heads and no tails, we would be estimating ^p=1 which is obviously a bad estimate: it implies that we're sure the coin has a probability 1 of coming up heads, which seems like an extreme inference to make from a sequence of N heads. After all, if we merely had p<1/(2N), we would not expect to see any heads to begin with. An inference that goes beyond this bound seems intuitively unjustified.

The simple version of Laplace's rule is a heuristic which says we should pretend we got one heads and one tails in excess of the results we have actually obtained and compute the naive estimator under that assumption. In other words, we would estimate ^p=(H+1)/(H+T+2). This avoids the pitfall mentioned in the previous paragraph even though it is a biased estimator: for N tosses we have

E[^p]=pN+1N+2=p(N+2)+(1−2p)N+2=p+1−2pN+2

which is not equal to the true value p unless p=1/2. This is unavoidable, because if we want to avoid overconfidence in the case of N heads, we have to make the estimator biased in the case p=1; and by continuity it will also at least be biased in some neighborhood of 1. It is, however, asymptotically unbiased: this means that as N→∞, we have E[^p]→p.

It's easy to see why this is an improvement over the naive estimator for avoiding overconfidence, but it looks more like an ad-hoc fix than an approach with theoretical justification. To give the justification behind Laplace's rule, I first have to talk about conjugate priors.

Conjugate priors

In Bayesian inference, we often have the following problem: there is a family of models we're considering that are parametrized by a vector θ∈Rn of parameters and we have a prior P(θ) over these parameters. Given the prior, we observe some event X, and we want to do Bayesian updating to get a posterior by

P(θ|X)∝P(θ)⋅P(X|θ)

Here ∝ is the proportionality symbol, since I'm omitting the overall normalizing factor P(X) in Bayes' rule for simplicity.

In general this becomes a mess: the posterior P(θ|X) can become intractable even after a simple update like this, since in principle there's no reason for the prior P(θ) and the likelihood P(X|θ) to have anything to do with each other. To simplify Bayesian inference in cases like this, a common trick is to pick a family of priors given the form of the likelihood function such that the posterior will also lie in the same family as the prior. Given the likelihood function, such families of priors are called conjugate priors.

Suppose we try to do this for our coin setup. We take θ=p, and the likelihood function of observing a specific sequence of H heads and T tails will then be pH(1−p)T. The prior and posterior will be related by

P(p|X)∝P(p)⋅pH(1−p)T

It should be fairly obvious that we can consider a family of distributions Beta(α,β)∝pα−1(1−p)β−1 in this case. Such density functions can be normalized to give genuine probability densities as long as α,β>0. If the prior P(p) is of this form, all the Bayesian update will do is change its exponents: we'll go from a prior of the form Beta(α,β) to a posterior of the form Beta(α+H,β+T). Thus, as long as our initial prior is of thi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Laplace's rule of succession, published by Ege Erdil on November 23, 2021 on LessWrong.

Laplace's rule of succession is a commonly used heuristic in forecasting the occurence of events which have little to no historical precedents. The basic problem setup is this: suppose you flip a coin with an unknown bias N=H+T times and get H heads and T tails. We want to infer something about the bias p of the coin from this information, where we define p to be the probability of the coin coming up heads.

One obvious idea is to estimate p by ^p=H/(H+T). This works well if we have some decent number of heads and tails, but it fails badly in the edge case where the tosses are very imbalanced. In the extreme case, if we get all heads and no tails, we would be estimating ^p=1 which is obviously a bad estimate: it implies that we're sure the coin has a probability 1 of coming up heads, which seems like an extreme inference to make from a sequence of N heads. After all, if we merely had p<1/(2N), we would not expect to see any heads to begin with. An inference that goes beyond this bound seems intuitively unjustified.

The simple version of Laplace's rule is a heuristic which says we should pretend we got one heads and one tails in excess of the results we have actually obtained and compute the naive estimator under that assumption. In other words, we would estimate ^p=(H+1)/(H+T+2). This avoids the pitfall mentioned in the previous paragraph even though it is a biased estimator: for N tosses we have

E[^p]=pN+1N+2=p(N+2)+(1−2p)N+2=p+1−2pN+2

which is not equal to the true value p unless p=1/2. This is unavoidable, because if we want to avoid overconfidence in the case of N heads, we have to make the estimator biased in the case p=1; and by continuity it will also at least be biased in some neighborhood of 1. It is, however, asymptotically unbiased: this means that as N→∞, we have E[^p]→p.

It's easy to see why this is an improvement over the naive estimator for avoiding overconfidence, but it looks more like an ad-hoc fix than an approach with theoretical justification. To give the justification behind Laplace's rule, I first have to talk about conjugate priors.

Conjugate priors

In Bayesian inference, we often have the following problem: there is a family of models we're considering that are parametrized by a vector θ∈Rn of parameters and we have a prior P(θ) over these parameters. Given the prior, we observe some event X, and we want to do Bayesian updating to get a posterior by

P(θ|X)∝P(θ)⋅P(X|θ)

Here ∝ is the proportionality symbol, since I'm omitting the overall normalizing factor P(X) in Bayes' rule for simplicity.

In general this becomes a mess: the posterior P(θ|X) can become intractable even after a simple update like this, since in principle there's no reason for the prior P(θ) and the likelihood P(X|θ) to have anything to do with each other. To simplify Bayesian inference in cases like this, a common trick is to pick a family of priors given the form of the likelihood function such that the posterior will also lie in the same family as the prior. Given the likelihood function, such families of priors are called conjugate priors.

Suppose we try to do this for our coin setup. We take θ=p, and the likelihood function of observing a specific sequence of H heads and T tails will then be pH(1−p)T. The prior and posterior will be related by

P(p|X)∝P(p)⋅pH(1−p)T

It should be fairly obvious that we can consider a family of distributions Beta(α,β)∝pα−1(1−p)β−1 in this case. Such density functions can be normalized to give genuine probability densities as long as α,β>0. If the prior P(p) is of this form, all the Bayesian update will do is change its exponents: we'll go from a prior of the form Beta(α,β) to a posterior of the form Beta(α+H,β+T). Thus, as long as our initial prior is of thi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Red-Team Against the Impact of Small Donations, published by AppliedDivinityStudies on November 24, 2021 on The Effective Altruism Forum.

In a comment on Benjamin Todd's article in favor of small donors, NunoSempere writes:

This article is kind of too "feel good" for my tastes. I'd also like to see a more angsty post that tries to come to grips with the fact that most of the impact is most likely not going to come from the individual people, and tries to see if this has any new implications, rather than justifying that all is good.

I am naturally an angsty person, and I don't carry much reputational risk, so this seemed like a natural fit.

I agree with NunoSempere that Benjamin's epistemics might be suffering from the nobility of his message. It's a feel-good encouragement to give, complete with a sympathetic photo of a very poor person who might benefit from your generosity. Because that message is so good and important, it requires a different style of writing and thinking than "let's try very hard to figure out what's true."

Additionally, I see Benjamin's post as a reaction to some popular myths. This is great, but we shouldn't mistake "some arguments against X are wrong" for "X is correct".

As to not bury the lede: I think there are better uses of your time than earning-to-give. Specifically, you ought to do more entrepreneurial, risky, and hyper-ambitious direct work, while simultaneously considering weirder and more speculative small donations.

Funny enough, although this is framed as a "red-team" post, I think that Benjamin mostly agrees with that advice. You can choose to take this as evidence that the advice is robust to worldview diversification, or as evidence that I'm really bad at red-teaming and falling prey to justification drift.

In terms of epistemic status: I take my own arguments here seriously, but I don't see them as definitive. Specifically, this post is meant to counterbalance Benjamin's post, so you should read his first, or at least read it later as a counterbalance against this one.

1. Our default view should be that high-impact funding capacity is already filled.

Consider Benjamin's explanation for why donating to LTFF is so valuable:

I would donate to the Long Term Future Fund over the global health fund, and would expect it to be perhaps 10-100x more cost-effective (and donating to global health is already very good). This is mainly because I think issues like AI safety and global catastrophic biorisks are bigger in scale and more neglected than global health.

I absolutely agree that those issues are very neglected, but only among the general population. They're not at all neglected within EA. Specifically, the question we should be asking isn't "do people care enough about this", but "how far will my marginal dollar go?"

To answer that latter question, it's not enough to highlight the importance of the issue, you would have to argue that:

There are longtermist organizations that are currently funding-constrained,

Such that more funding would enable them to do more or better work,

And this funding can't be met by existing large EA philanthropists.

It's not clear to me that any of these points are true. They might be, but Benjamin doesn't take the time to argue for them very rigorously. Lacking strong evidence, my default assumptions are that funding capacity for extremely high-impact organizations well aligned with EA ideology will be filled by donors.

Benjamin does admirably clarify that there are specific programs he has in mind:

there are ways that longtermists could deploy billions of dollars and still do a significant amount of good. For instance, CEPI is a $3.5bn programme to develop vaccines to fight the next pandemic.

At face value, CEPI seems great. But at the meta-level, I still have to ask, if CEPI is a good use of funds, why...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Red-Team Against the Impact of Small Donations, published by AppliedDivinityStudies on November 24, 2021 on The Effective Altruism Forum.

In a comment on Benjamin Todd's article in favor of small donors, NunoSempere writes:

This article is kind of too "feel good" for my tastes. I'd also like to see a more angsty post that tries to come to grips with the fact that most of the impact is most likely not going to come from the individual people, and tries to see if this has any new implications, rather than justifying that all is good.

I am naturally an angsty person, and I don't carry much reputational risk, so this seemed like a natural fit.

I agree with NunoSempere that Benjamin's epistemics might be suffering from the nobility of his message. It's a feel-good encouragement to give, complete with a sympathetic photo of a very poor person who might benefit from your generosity. Because that message is so good and important, it requires a different style of writing and thinking than "let's try very hard to figure out what's true."

Additionally, I see Benjamin's post as a reaction to some popular myths. This is great, but we shouldn't mistake "some arguments against X are wrong" for "X is correct".

As to not bury the lede: I think there are better uses of your time than earning-to-give. Specifically, you ought to do more entrepreneurial, risky, and hyper-ambitious direct work, while simultaneously considering weirder and more speculative small donations.

Funny enough, although this is framed as a "red-team" post, I think that Benjamin mostly agrees with that advice. You can choose to take this as evidence that the advice is robust to worldview diversification, or as evidence that I'm really bad at red-teaming and falling prey to justification drift.

In terms of epistemic status: I take my own arguments here seriously, but I don't see them as definitive. Specifically, this post is meant to counterbalance Benjamin's post, so you should read his first, or at least read it later as a counterbalance against this one.

1. Our default view should be that high-impact funding capacity is already filled.

Consider Benjamin's explanation for why donating to LTFF is so valuable:

I would donate to the Long Term Future Fund over the global health fund, and would expect it to be perhaps 10-100x more cost-effective (and donating to global health is already very good). This is mainly because I think issues like AI safety and global catastrophic biorisks are bigger in scale and more neglected than global health.

I absolutely agree that those issues are very neglected, but only among the general population. They're not at all neglected within EA. Specifically, the question we should be asking isn't "do people care enough about this", but "how far will my marginal dollar go?"

To answer that latter question, it's not enough to highlight the importance of the issue, you would have to argue that:

There are longtermist organizations that are currently funding-constrained,

Such that more funding would enable them to do more or better work,

And this funding can't be met by existing large EA philanthropists.

It's not clear to me that any of these points are true. They might be, but Benjamin doesn't take the time to argue for them very rigorously. Lacking strong evidence, my default assumptions are that funding capacity for extremely high-impact organizations well aligned with EA ideology will be filled by donors.

Benjamin does admirably clarify that there are specific programs he has in mind:

there are ways that longtermists could deploy billions of dollars and still do a significant amount of good. For instance, CEPI is a $3.5bn programme to develop vaccines to fight the next pandemic.

At face value, CEPI seems great. But at the meta-level, I still have to ask, if CEPI is a good use of funds, why...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [AN #169]: Collaborating with humans without human data, published by Rohin Shah on November 24, 2021 on The AI Alignment Forum.

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

SECTIONS

HIGHLIGHTS

This paper shows that this can be done with Fictitious Co-Play (FCP), in which we train our final agent against a population of self-play agents and their past checkpoints taken throughout training. Such agents get significantly higher rewards when collaborating with humans in Overcooked (relative to the human-aware approach in the previously linked paper).

In their ablations, the authors find that it is particularly important to include past checkpoints in the population against which you train. They also test whether it helps to have the self-play agents have a variety of architectures, and find that it mostly does not make a difference (as long as you are using past checkpoints as well).

Read more: Related paper: Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Rohin's opinion: You could imagine two different philosophies on how to build AI systems -- the first option is to train them on the actual task of interest (for Overcooked, training agents to play against humans or human models), while the second option is to train a more robust agent on some more general task that hopefully includes the actual task within it (the approach in this paper). Besides Overcooked, another example would be supervised learning on some natural language task (the first philosophy), as compared to pretraining on the Internet GPT-style and then prompting the model to solve your task of interest (the second philosophy). In some sense the quest for a single unified AGI system is itself a bet on the second philosophy -- first you build your AGI that can do all tasks, and then you point it at the specific task you want to do now.

Historically, I think AI has focused primarily on the first philosophy, but recent years have shown the power of the second philosophy. However, I don’t think the question is settled yet: one issue with the second philosophy is that it is often difficult to fully “aim” your system at the true task of interest, and as a result it doesn’t perform as well as it “could have”. In Overcooked, the FCP agents will not learn specific quirks of human gameplay that could be exploited to improve efficiency (which the human-aware agent could do, at least in theory). In natural language, even if you prompt GPT-3 appropriately, there’s still some chance it ends up rambling about something else entirely, or neglects to mention some information that it “knows” but that a human on the Internet would not have said. (See also this post (AN #141).)

I should note that you can also have a hybrid approach, where you start by training a large model with the second philosophy, and then you finetune it on your task of interest as in the first philosophy, gaining the benefits of both.

I’m generally interested in which approach will build more useful agents, as this seems quite relevant to forecasting the future of AI (which in turn affects lots of things including AI alignment plans).

TECHNICAL AI ALIGNMENT

LEARNING HUMAN INTENT

The paper provides a mathematical treatment of both the forward problem (how to compute actions in the planner given the reward, think of algorithms like value iteration) and the backward problem (how to compute the reward given demonstrations, the typical...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [AN #169]: Collaborating with humans without human data, published by Rohin Shah on November 24, 2021 on The AI Alignment Forum.

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

SECTIONS

HIGHLIGHTS

This paper shows that this can be done with Fictitious Co-Play (FCP), in which we train our final agent against a population of self-play agents and their past checkpoints taken throughout training. Such agents get significantly higher rewards when collaborating with humans in Overcooked (relative to the human-aware approach in the previously linked paper).

In their ablations, the authors find that it is particularly important to include past checkpoints in the population against which you train. They also test whether it helps to have the self-play agents have a variety of architectures, and find that it mostly does not make a difference (as long as you are using past checkpoints as well).

Read more: Related paper: Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Rohin's opinion: You could imagine two different philosophies on how to build AI systems -- the first option is to train them on the actual task of interest (for Overcooked, training agents to play against humans or human models), while the second option is to train a more robust agent on some more general task that hopefully includes the actual task within it (the approach in this paper). Besides Overcooked, another example would be supervised learning on some natural language task (the first philosophy), as compared to pretraining on the Internet GPT-style and then prompting the model to solve your task of interest (the second philosophy). In some sense the quest for a single unified AGI system is itself a bet on the second philosophy -- first you build your AGI that can do all tasks, and then you point it at the specific task you want to do now.

Historically, I think AI has focused primarily on the first philosophy, but recent years have shown the power of the second philosophy. However, I don’t think the question is settled yet: one issue with the second philosophy is that it is often difficult to fully “aim” your system at the true task of interest, and as a result it doesn’t perform as well as it “could have”. In Overcooked, the FCP agents will not learn specific quirks of human gameplay that could be exploited to improve efficiency (which the human-aware agent could do, at least in theory). In natural language, even if you prompt GPT-3 appropriately, there’s still some chance it ends up rambling about something else entirely, or neglects to mention some information that it “knows” but that a human on the Internet would not have said. (See also this post (AN #141).)

I should note that you can also have a hybrid approach, where you start by training a large model with the second philosophy, and then you finetune it on your task of interest as in the first philosophy, gaining the benefits of both.

I’m generally interested in which approach will build more useful agents, as this seems quite relevant to forecasting the future of AI (which in turn affects lots of things including AI alignment plans).

TECHNICAL AI ALIGNMENT

LEARNING HUMAN INTENT

The paper provides a mathematical treatment of both the forward problem (how to compute actions in the planner given the reward, think of algorithms like value iteration) and the backward problem (how to compute the reward given demonstrations, the typical...