Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #6, published by Zvi on December 13, 2021 on LessWrong.

Previous posts: #1, #2, #3, #4, #5, last weekly post

Well, that escalated quickly. Omicron has already escalated quickly in log terms, and will soon escalate quickly in linear terms. The core questions remain the same, and we continue to continuously get better data to allow us to make better guesses and projections.

There are three central places where population-level information is available first:

South Africa, because Omicron has taken over already and they have good data.

Denmark, because they do all the sequencing.

United Kingdom, because of S-gene-deletion data and good sequencing.

We also have a few African countries like Zimbabwe with giant spikes. But in other European countries and in the United States, the lack of sequencing makes it impossible to know how far along things are or how fast they are moving.

Thus, we must rely on what data we have, and give thanks to those who provide it.

Uncertainty abounds but there is no known link between ‘has more Omicron infections’ and ‘has ability and willingness to monitor Omicron cases’ so we should start with the assumption that Denmark and the United Kingdom are typical western countries for this, adjusted for the state of Covid-19 under Delta there relative to other places.

All of that can and should be complemented by experimentation and lab results. Alas, as per usual, challenge trials and most non-natural experiments remain illegal, so we have orders of magnitude less knowledge and certainty from these sources, and everyone is much worse off, than we could if we cared more about saving lives and making people’s lives better and actual ethics, and prioritized those considerations over ‘biomedical ethics.’ So mostly the useful findings lie elsewhere.

Here’s two charts of overall cases and positive test percentages in various places, for context. The rise in Denmark does reflect some Omicron cases, they’d still be under 1,000 on this graph without it, and the UK rise as well.

South Africa

The most hopeful news would be if South African cases peaked all of a sudden for no clear reason, the same way cases of Delta in India did, and started heading downward. We did see a blip for a few days, but was it more than a blip? My read on it is no, and there are signs of issues with data reporting to explain it.

The news on this front is still good, because it’s possible things are improving, and in the worst scenarios things should have continued to get rapidly worse. The positivity rate sticking around 30% would be actively great news.

As you can see on the graph above, the 7-day averages haven’t stopped rising, at least once Our World In Data corrected for temporary data errors. I do still find the lack of a steeper rise to be good news.

Here’s a short explanation of why it’s still too early to draw any conclusions from the death rate. Still need another week or so due to lag.

There are a bunch of threads about South Africa in the Threads section below. They basically all note that a lot of cases continue to be incidental and severity looks low, but that it’s too early to know for sure for reasons I went into on Thursday.

Here’s a video interview with Dr. Angelique Coetzee of the South African Medical Association from December 12. Most optimistic case presented so far. I want to update more but I still don’t feel like this clarifies things much.

Denmark

Denmark has the world’s best sequencing. It also has the largest number of known Omicron cases as a share of population by a lot, which is not a coincidence. And it knows cases are growing exponentially, also not a coincidence.

Here’s an update from yesterday.

Arxiv link.

For speed reasons I didn’t double check the math, but the principles are right and the answer is roughly correct. This is happening,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #6, published by Zvi on December 13, 2021 on LessWrong.

Previous posts: #1, #2, #3, #4, #5, last weekly post

Well, that escalated quickly. Omicron has already escalated quickly in log terms, and will soon escalate quickly in linear terms. The core questions remain the same, and we continue to continuously get better data to allow us to make better guesses and projections.

There are three central places where population-level information is available first:

South Africa, because Omicron has taken over already and they have good data.

Denmark, because they do all the sequencing.

United Kingdom, because of S-gene-deletion data and good sequencing.

We also have a few African countries like Zimbabwe with giant spikes. But in other European countries and in the United States, the lack of sequencing makes it impossible to know how far along things are or how fast they are moving.

Thus, we must rely on what data we have, and give thanks to those who provide it.

Uncertainty abounds but there is no known link between ‘has more Omicron infections’ and ‘has ability and willingness to monitor Omicron cases’ so we should start with the assumption that Denmark and the United Kingdom are typical western countries for this, adjusted for the state of Covid-19 under Delta there relative to other places.

All of that can and should be complemented by experimentation and lab results. Alas, as per usual, challenge trials and most non-natural experiments remain illegal, so we have orders of magnitude less knowledge and certainty from these sources, and everyone is much worse off, than we could if we cared more about saving lives and making people’s lives better and actual ethics, and prioritized those considerations over ‘biomedical ethics.’ So mostly the useful findings lie elsewhere.

Here’s two charts of overall cases and positive test percentages in various places, for context. The rise in Denmark does reflect some Omicron cases, they’d still be under 1,000 on this graph without it, and the UK rise as well.

South Africa

The most hopeful news would be if South African cases peaked all of a sudden for no clear reason, the same way cases of Delta in India did, and started heading downward. We did see a blip for a few days, but was it more than a blip? My read on it is no, and there are signs of issues with data reporting to explain it.

The news on this front is still good, because it’s possible things are improving, and in the worst scenarios things should have continued to get rapidly worse. The positivity rate sticking around 30% would be actively great news.

As you can see on the graph above, the 7-day averages haven’t stopped rising, at least once Our World In Data corrected for temporary data errors. I do still find the lack of a steeper rise to be good news.

Here’s a short explanation of why it’s still too early to draw any conclusions from the death rate. Still need another week or so due to lag.

There are a bunch of threads about South Africa in the Threads section below. They basically all note that a lot of cases continue to be incidental and severity looks low, but that it’s too early to know for sure for reasons I went into on Thursday.

Here’s a video interview with Dr. Angelique Coetzee of the South African Medical Association from December 12. Most optimistic case presented so far. I want to update more but I still don’t feel like this clarifies things much.

Denmark

Denmark has the world’s best sequencing. It also has the largest number of known Omicron cases as a share of population by a lot, which is not a coincidence. And it knows cases are growing exponentially, also not a coincidence.

Here’s an update from yesterday.

Arxiv link.

For speed reasons I didn’t double check the math, but the principles are right and the answer is roughly correct. This is happening,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nines of safety: Terence Tao’s proposed unit of measurement of risk, published by anson.ho on December 12, 2021 on The Effective Altruism Forum.

I recently came across Terence Tao’s post proposing a risk measure called “nines of safety”. I think it’s a very interesting proposal, and given that many EA forum users think a lot about risks and probabilities, I’m curious to hear what opinions other people have.

Below I’ll briefly summarise my understanding of the idea, and ask some specific questions about how this might be related to EA. For more details, I highly recommend reading the original post.

The proposal

While we often use percentages to describe probabilities and proportions, it can be hard to tell whether a given percentage is “good” or “bad”. For instance, having a 60% chance of success seems risky for a medical operation, but a result of 60% to 40% would be a landslide victory in a two-party election.

Part of the difficulty is that percentages can be used in a multitude of ways, with different interpretations in different scenarios. Tao proposes a unit that can be used to measure both the risk and safety of some event that has a really bad outcome (e.g. a global pandemic). The trouble with percentages in this scenario is that the probability of getting a good outcome needs to be really high in order for us to be comfortable. For instance, 90% odds of successfully completing a potentially life-threatening medical operation might seem a bit risky. 99% odds would probably feel quite a bit better, and at 99.9% odds of success we might start feeling reasonably safe (in general, this depends on how bad the negative outcomes are, the counterfactual of not doing the operation, etc.).

Writing out all of these 9s seems a bit clumsy, and the measure that Tao proposes addresses this - it’s called “nines of safety”, and informally measures how many consecutive 9s there are in the probability of success (the nines of risk would be the same as the nines of safety, but applied to the probability of failure). So in the previous example:

90% success = 1 nine of safety

99% success = 2 nines of safety

99.9% success = 3 nines of safety

We can formalise this in terms of the base-10 logarithm:

which allows us to extend “nines of safety” so that it isn’t just a whole number. We can write a table to convert between the probabilities of success, failure, and the nines of safety:

0%

100%

0.0

50%

50%

0.3

75%

25%

0.6

80%

20%

0.7

90%

10%

1.0

95%

5%

1.3

97.5%

2.5%

1.6

98%

2%

1.7

99%

1%

2.0

99.5%

0.5%

2.3

99.75%

0.25%

2.6

99.8%

0.2%

2.7

99.9%

0.1%

3.0

99.95%

0.05%

3.3

99.975%

0.025%

3.6

99.98%

0.02%

3.7

99.99%

0.01%

4.0

100%

0%

infinite

Note that the nines of safety are rounded to 1 decimal place, because in practice probability estimates are likely to be quite uncertain, and extra decimal places may not be particularly significant.

In general (as in the aforementioned example of the medical operation), the number of nines of safety depends on several factors, such as the number of people exposed and the duration of exposure. We might also need to consider repeated exposures, which can be quite complicated – depending on the task, individual exposures may not necessarily be independent from each other.

Why?

This potentially has several benefits:

Easier mental arithmetic:

Due to the properties of logarithms, adding “nines of safety” is the same as multiplying probabilities, which makes mental calculation easier (especially true if we’re dealing with relative risks, e.g. “What are the odds of catching COVID in a vaccinated group relative to an unvaccinated control group?”)

It also makes it easier to convert from individual risk to group risk (see this comment)

“Apples-to-apples comparisons”: since percentages are interpreted very differently depended on context, having a measure that is uni...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nines of safety: Terence Tao’s proposed unit of measurement of risk, published by anson.ho on December 12, 2021 on The Effective Altruism Forum.

I recently came across Terence Tao’s post proposing a risk measure called “nines of safety”. I think it’s a very interesting proposal, and given that many EA forum users think a lot about risks and probabilities, I’m curious to hear what opinions other people have.

Below I’ll briefly summarise my understanding of the idea, and ask some specific questions about how this might be related to EA. For more details, I highly recommend reading the original post.

The proposal

While we often use percentages to describe probabilities and proportions, it can be hard to tell whether a given percentage is “good” or “bad”. For instance, having a 60% chance of success seems risky for a medical operation, but a result of 60% to 40% would be a landslide victory in a two-party election.

Part of the difficulty is that percentages can be used in a multitude of ways, with different interpretations in different scenarios. Tao proposes a unit that can be used to measure both the risk and safety of some event that has a really bad outcome (e.g. a global pandemic). The trouble with percentages in this scenario is that the probability of getting a good outcome needs to be really high in order for us to be comfortable. For instance, 90% odds of successfully completing a potentially life-threatening medical operation might seem a bit risky. 99% odds would probably feel quite a bit better, and at 99.9% odds of success we might start feeling reasonably safe (in general, this depends on how bad the negative outcomes are, the counterfactual of not doing the operation, etc.).

Writing out all of these 9s seems a bit clumsy, and the measure that Tao proposes addresses this - it’s called “nines of safety”, and informally measures how many consecutive 9s there are in the probability of success (the nines of risk would be the same as the nines of safety, but applied to the probability of failure). So in the previous example:

90% success = 1 nine of safety

99% success = 2 nines of safety

99.9% success = 3 nines of safety

We can formalise this in terms of the base-10 logarithm:

which allows us to extend “nines of safety” so that it isn’t just a whole number. We can write a table to convert between the probabilities of success, failure, and the nines of safety:

0%

100%

0.0

50%

50%

0.3

75%

25%

0.6

80%

20%

0.7

90%

10%

1.0

95%

5%

1.3

97.5%

2.5%

1.6

98%

2%

1.7

99%

1%

2.0

99.5%

0.5%

2.3

99.75%

0.25%

2.6

99.8%

0.2%

2.7

99.9%

0.1%

3.0

99.95%

0.05%

3.3

99.975%

0.025%

3.6

99.98%

0.02%

3.7

99.99%

0.01%

4.0

100%

0%

infinite

Note that the nines of safety are rounded to 1 decimal place, because in practice probability estimates are likely to be quite uncertain, and extra decimal places may not be particularly significant.

In general (as in the aforementioned example of the medical operation), the number of nines of safety depends on several factors, such as the number of people exposed and the duration of exposure. We might also need to consider repeated exposures, which can be quite complicated – depending on the task, individual exposures may not necessarily be independent from each other.

Why?

This potentially has several benefits:

Easier mental arithmetic:

Due to the properties of logarithms, adding “nines of safety” is the same as multiplying probabilities, which makes mental calculation easier (especially true if we’re dealing with relative risks, e.g. “What are the odds of catching COVID in a vaccinated group relative to an unvaccinated control group?”)

It also makes it easier to convert from individual risk to group risk (see this comment)

“Apples-to-apples comparisons”: since percentages are interpreted very differently depended on context, having a measure that is uni...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: External Evaluation of the EA Wiki, published by NunoSempere on December 13, 2021 on The Effective Altruism Forum.

tl;dr: The EA Wiki seems probably worth funding, but it is not the most ambitious project that the main person behind it could be doing.

Epistemic status: Although we have data and forecasts for some EA Wiki metrics, the most important variables—such as future growth or impact—require speculation. Readers should care or listen to that speculation insofar as they think that I have good judgment. But although we have significant uncertainties, we can still meaningfully evaluate the EA Wiki. In the future, we might be keen to apply this level of evaluation to other interventions.

This evaluation is a project by the Quantified Uncertainty Research Institute, commissioned by Pablo Stafforini and Max Daniel, financed by the EA Infrastructure Fund, and conducted by Nuño Sempere. Pablo and Max reviewed this document prior to publication.

Summary

The EA Wiki is a project which attempts to produce and organize EA content. It is hosted within the EA Forum on top of its tag system, and it is the latest attempt on a long line of EA Wiki projects which previously failed.

On the positive side, metrics like "words per dollar" and "views per dollar" are high. Also, Pablo Stafforini—the EA Wiki’s current lead—is a good fit for the project and is very prolific.

On the negative side, Pablo alone spends comparable amounts working on the wiki as users cumulatively spend reading it. The EA Wiki—and in particular the editing festival—doesn't seem to have resulted in junior EAs doing many edits to the Wiki, suggesting that its movement-building-related benefits are on the low side. Further, the wiki doesn’t seem that useful right now in terms of helping current research efforts, such as those of EA research organizations or independent researchers.

On the neutral side, it seems like it will take more time—perhaps on the order of a few years—to determine whether the EA Wiki is successful or not. The case for the EA Wiki right now relies on its increased future usefulness. The wiki is early, so future software functionality and content might make existing content more valuable.

The fact that we need more time to know the influence of the wiki shouldn’t be particularly surprising. Given that similar projects previously failed, it would be surprising if this iteration immediately succeeded.

Some projects could be built on top of the EA Wiki, like more detailed and granular hierarchies of concepts, an accessible curated repository of EA forum posts with links and collected references, or more in-depth articles rather than stubs. Some more examples of projects are given in their own section below. I would feel more excited about the EA Wiki exploring some of these adjacent projects, rather than just continuing to host very short articles (stubs) as it does now.

Overall, I conclude that the EA Wiki seems worth funding for CEA. For Pablo himself, I think he might be able to find more ambitious work to do sometime in the future, but I recognize that this could be difficult.

Introduction

Background

The purpose of this evaluation was.

Provide evidence and opinion to inform

whether Pablo Stafforini should decide to continue working on this project

whether the EA Infrastructure Fund should renew Pablo Stafforini’s grant

If Pablo does continue to work on the EA wiki, make suggestions on how this could create more value.

As a frequent participant in the EA Forum, I (Nuño) began this evaluation with a skeptical to a negative impression of the EA Wiki. I found the tagging system useful, but I had otherwise not used the Wiki much myself.

Stated goals of the EA Wiki

Per the two grant rationales evaluating the EA Wiki’s expected impact in April 2020 and June 2021, the two main hopes for it were:

1....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: External Evaluation of the EA Wiki, published by NunoSempere on December 13, 2021 on The Effective Altruism Forum.

tl;dr: The EA Wiki seems probably worth funding, but it is not the most ambitious project that the main person behind it could be doing.

Epistemic status: Although we have data and forecasts for some EA Wiki metrics, the most important variables—such as future growth or impact—require speculation. Readers should care or listen to that speculation insofar as they think that I have good judgment. But although we have significant uncertainties, we can still meaningfully evaluate the EA Wiki. In the future, we might be keen to apply this level of evaluation to other interventions.

This evaluation is a project by the Quantified Uncertainty Research Institute, commissioned by Pablo Stafforini and Max Daniel, financed by the EA Infrastructure Fund, and conducted by Nuño Sempere. Pablo and Max reviewed this document prior to publication.

Summary

The EA Wiki is a project which attempts to produce and organize EA content. It is hosted within the EA Forum on top of its tag system, and it is the latest attempt on a long line of EA Wiki projects which previously failed.

On the positive side, metrics like "words per dollar" and "views per dollar" are high. Also, Pablo Stafforini—the EA Wiki’s current lead—is a good fit for the project and is very prolific.

On the negative side, Pablo alone spends comparable amounts working on the wiki as users cumulatively spend reading it. The EA Wiki—and in particular the editing festival—doesn't seem to have resulted in junior EAs doing many edits to the Wiki, suggesting that its movement-building-related benefits are on the low side. Further, the wiki doesn’t seem that useful right now in terms of helping current research efforts, such as those of EA research organizations or independent researchers.

On the neutral side, it seems like it will take more time—perhaps on the order of a few years—to determine whether the EA Wiki is successful or not. The case for the EA Wiki right now relies on its increased future usefulness. The wiki is early, so future software functionality and content might make existing content more valuable.

The fact that we need more time to know the influence of the wiki shouldn’t be particularly surprising. Given that similar projects previously failed, it would be surprising if this iteration immediately succeeded.

Some projects could be built on top of the EA Wiki, like more detailed and granular hierarchies of concepts, an accessible curated repository of EA forum posts with links and collected references, or more in-depth articles rather than stubs. Some more examples of projects are given in their own section below. I would feel more excited about the EA Wiki exploring some of these adjacent projects, rather than just continuing to host very short articles (stubs) as it does now.

Overall, I conclude that the EA Wiki seems worth funding for CEA. For Pablo himself, I think he might be able to find more ambitious work to do sometime in the future, but I recognize that this could be difficult.

Introduction

Background

The purpose of this evaluation was.

Provide evidence and opinion to inform

whether Pablo Stafforini should decide to continue working on this project

whether the EA Infrastructure Fund should renew Pablo Stafforini’s grant

If Pablo does continue to work on the EA wiki, make suggestions on how this could create more value.

As a frequent participant in the EA Forum, I (Nuño) began this evaluation with a skeptical to a negative impression of the EA Wiki. I found the tagging system useful, but I had otherwise not used the Wiki much myself.

Stated goals of the EA Wiki

Per the two grant rationales evaluating the EA Wiki’s expected impact in April 2020 and June 2021, the two main hopes for it were:

1....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solving Interpretability Week, published by elriggs on December 13, 2021 on The AI Alignment Forum.

For original motivation, see solving corrigibility week. I'll state what I learned from corrigibility week, why I didn't post one last week, and the updated format for interpretability.

What I Noticed

The open call to co-work with my calendly link didn't work. Only met with people when I messaged them specifically. I'm changing my availability from "all day Saturday" to "two hours/day" to better meet other's schedule and not have a single point of failure (see next section). It also seems great to ask specific people to meet when I have a question on their research.

Going through previous work, it was good to write out my thoughts in the google doc, spruce it up, and make it a comment on the original post. This also connects with messaging people to meet with if I feel there's a large disconnect or I have a lot of confusion regarding their work.

The google doc was also less long-term collaborative, which may have been from a lack of notifications that someone responded to what you wrote. So I'm moving the research direction part to the comment section here.

No Post Last Week

I committed to posting 3 of these this month, but didn't last week. I had a bad Saturday (which was my designate work day for the corrigibility post) and a bad week until I talked to Shay, my therapist, and worked out a plan (Shay is great and has free sessions for those working on alignment; highly recommend). I am not ashamed or embarrassed, but I do put less stock in my public commitments and more aware of my failure modes. I personally still recommend people try (and possibly fail) ambitious projects.

I do want to write a post on corrigibility, but it's too time-consuming to both work out my own thoughts going through the literature and meet with people to understand their work & distill those conversations. Both are important and not mutually exclusive. Some possible solutions are to circle back to corrigibility next year or take two weeks per topic.

Format This Week

Here is the google doc for interpretability. In the comments are top-level comments for:

Research directions you want discussed with any questions you have

Your meeting schedule for co-working and how you'd like to be contacted

Suggestions for future weeks or changes to the format

In the doc are:

Literature Review

Tasks to do for further research

Again, in the google doc, it is socially acceptable to write low-quality babble. In this post's comments, I also accept babbling/ spit-balling and will not delete them.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solving Interpretability Week, published by elriggs on December 13, 2021 on The AI Alignment Forum.

For original motivation, see solving corrigibility week. I'll state what I learned from corrigibility week, why I didn't post one last week, and the updated format for interpretability.

What I Noticed

The open call to co-work with my calendly link didn't work. Only met with people when I messaged them specifically. I'm changing my availability from "all day Saturday" to "two hours/day" to better meet other's schedule and not have a single point of failure (see next section). It also seems great to ask specific people to meet when I have a question on their research.

Going through previous work, it was good to write out my thoughts in the google doc, spruce it up, and make it a comment on the original post. This also connects with messaging people to meet with if I feel there's a large disconnect or I have a lot of confusion regarding their work.

The google doc was also less long-term collaborative, which may have been from a lack of notifications that someone responded to what you wrote. So I'm moving the research direction part to the comment section here.

No Post Last Week

I committed to posting 3 of these this month, but didn't last week. I had a bad Saturday (which was my designate work day for the corrigibility post) and a bad week until I talked to Shay, my therapist, and worked out a plan (Shay is great and has free sessions for those working on alignment; highly recommend). I am not ashamed or embarrassed, but I do put less stock in my public commitments and more aware of my failure modes. I personally still recommend people try (and possibly fail) ambitious projects.

I do want to write a post on corrigibility, but it's too time-consuming to both work out my own thoughts going through the literature and meet with people to understand their work & distill those conversations. Both are important and not mutually exclusive. Some possible solutions are to circle back to corrigibility next year or take two weeks per topic.

Format This Week

Here is the google doc for interpretability. In the comments are top-level comments for:

Research directions you want discussed with any questions you have

Your meeting schedule for co-working and how you'd like to be contacted

Suggestions for future weeks or changes to the format

In the doc are:

Literature Review

Tasks to do for further research

Again, in the google doc, it is socially acceptable to write low-quality babble. In this post's comments, I also accept babbling/ spit-balling and will not delete them.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Redwood's Technique-Focused Epistemic Strategy, published by adamShimi on December 12, 2021 on LessWrong.

Imagine you’re part of a team of ML engineers and research scientists, and you want to help with alignment. Everyone is ready to jump in the fray; there’s only one problem — how are you supposed to do applied research when you don’t really know how AGI will be built, what it will look like, not even the architecture or something like that? What you have is the current state of ML, and a lot of conceptual and theoretical arguments.

You’re in dire need of a bridge (an epistemic strategy) between the experiments you could run now and the knowledge that will serve for solving alignment.

Redwood Research is in this precise situation. And they have a bridge in mind. Hence this post, where I write down my interpretation of their approach, based on conversations with Redwood’s Buck Shlegeris. As such, even if I’m describing Redwood’s strategy, it’s probably biased towards what sounds most relevant to both Buck and me.

Thanks to Buck Shlegeris for great discussions and feedback. Thanks to Nate Thomas for listening to me when we talked about epistemic strategies and for pushing me to send him and Buck my draft, starting this whole collaboration. Thanks to Seraphina Nix for feedback on a draft of this post.

Techniques, not Tasks or Issues

My first intuition, when thinking about such a bridge between modern experiments and useful practical knowledge for alignment, is to focus on tasks and/or issues. By task, I mean the sort of things we want an aligned AGI to do (“learn what killing means and not do it” might be an example), whereas issues are. issues with an AGI (deception for example). It sounded obvious to me that you start from one of those, and then try to make a simpler, analogous version you can solve with modern technology— the trick being how to justify your analogy.

This is not at all how Buck sees it. After my initial confusion wore off, I realized he thinks in terms of techniques: potential ways of aligning an AGI. If tasks are “what we want” and issues are “what might go wrong”, techniques focus on “how” — how we solve the task and avoid the issues.

Buck’s favorite example of a technique (and the one driving Redwood’s current work) is a form of adversarial training where the main model receives adversarial examples from a trained adversary, and has its response judged by an overseer to see if they’re acceptable or not.

Another example is Debate: the alignment proposal where two models debate on the questions proposed by the human judge/supervisor, and hopefully the honest strategy is favored such that it should always win the debate.

Have you seen the trend with these two examples? They both can be specified as ML problems. What I mean is that the researchers who invented these techniques provided broad strokes ML setups and parameterized loss functions for them. Of course, we can’t train all the relevant parts yet (adversary and overseer in adversarial training, advanced debaters in debate). If we could, then studying the technique would just boil down to straightforward ML work.

Yet these two ingredients, ML specifications and hardness of specific parts, point to Redwood’s bridge between their experiments and the AGI-level technique.

Aside: Choosing a Technique

Before going into the detail of the bridge, I want to clarify that I’m not going to discuss how to choose a technique. That depends on many subtle arguments and discussions, especially conceptual ones. This is an expertise that Redwood partly has already and that they want to improve on. But at the moment, the work they’ve done focuses on turning a technique they’ve chosen into relevant experiments they can run today. Hence the focus of this post.

Replacing AGI-level Capabilities

Recall that the main re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Redwood's Technique-Focused Epistemic Strategy, published by adamShimi on December 12, 2021 on LessWrong.

Imagine you’re part of a team of ML engineers and research scientists, and you want to help with alignment. Everyone is ready to jump in the fray; there’s only one problem — how are you supposed to do applied research when you don’t really know how AGI will be built, what it will look like, not even the architecture or something like that? What you have is the current state of ML, and a lot of conceptual and theoretical arguments.

You’re in dire need of a bridge (an epistemic strategy) between the experiments you could run now and the knowledge that will serve for solving alignment.

Redwood Research is in this precise situation. And they have a bridge in mind. Hence this post, where I write down my interpretation of their approach, based on conversations with Redwood’s Buck Shlegeris. As such, even if I’m describing Redwood’s strategy, it’s probably biased towards what sounds most relevant to both Buck and me.

Thanks to Buck Shlegeris for great discussions and feedback. Thanks to Nate Thomas for listening to me when we talked about epistemic strategies and for pushing me to send him and Buck my draft, starting this whole collaboration. Thanks to Seraphina Nix for feedback on a draft of this post.

Techniques, not Tasks or Issues

My first intuition, when thinking about such a bridge between modern experiments and useful practical knowledge for alignment, is to focus on tasks and/or issues. By task, I mean the sort of things we want an aligned AGI to do (“learn what killing means and not do it” might be an example), whereas issues are. issues with an AGI (deception for example). It sounded obvious to me that you start from one of those, and then try to make a simpler, analogous version you can solve with modern technology— the trick being how to justify your analogy.

This is not at all how Buck sees it. After my initial confusion wore off, I realized he thinks in terms of techniques: potential ways of aligning an AGI. If tasks are “what we want” and issues are “what might go wrong”, techniques focus on “how” — how we solve the task and avoid the issues.

Buck’s favorite example of a technique (and the one driving Redwood’s current work) is a form of adversarial training where the main model receives adversarial examples from a trained adversary, and has its response judged by an overseer to see if they’re acceptable or not.

Another example is Debate: the alignment proposal where two models debate on the questions proposed by the human judge/supervisor, and hopefully the honest strategy is favored such that it should always win the debate.

Have you seen the trend with these two examples? They both can be specified as ML problems. What I mean is that the researchers who invented these techniques provided broad strokes ML setups and parameterized loss functions for them. Of course, we can’t train all the relevant parts yet (adversary and overseer in adversarial training, advanced debaters in debate). If we could, then studying the technique would just boil down to straightforward ML work.

Yet these two ingredients, ML specifications and hardness of specific parts, point to Redwood’s bridge between their experiments and the AGI-level technique.

Aside: Choosing a Technique

Before going into the detail of the bridge, I want to clarify that I’m not going to discuss how to choose a technique. That depends on many subtle arguments and discussions, especially conceptual ones. This is an expertise that Redwood partly has already and that they want to improve on. But at the moment, the work they’ve done focuses on turning a technique they’ve chosen into relevant experiments they can run today. Hence the focus of this post.

Replacing AGI-level Capabilities

Recall that the main re...