Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Peak defense vs trough defense in biosecurity, published by ASB on December 16, 2021 on The Effective Altruism Forum.

Many biosecurity interventions (e.g. faster vaccine manufacturing, strengthening the biological weapons convention) are useful against a wide category of biological risks, including the extreme worst cases as well as less extreme risks. However, it is worth highlighting areas where this is not the case, since things that are useful primarily for extreme worst-case risks might be especially neglected (and when I talk about “extreme worst-case risks,” I mean the kind that could threaten permanent civilizational collapse and would be especially important under a longtermist framework).

Some interventions with this property might look something like ‘protect a small population at all costs,’ with the canonical example being bunkers (which does nothing for the vast majority of current people, but provides extra protection for a small and lucky handful). If we assume that civilization could ultimately recover even with a relatively small number of survivors, these interventions would still be worthwhile (for more on why this assumption is likely true, see here).

Many other areas of biosecurity could look substantially different under a strategy that prioritizes survival in the most extreme scenarios. A few illustrative examples:

Medical countermeasures (e.g. vaccines): for prioritizing survival, one might focus on ensuring a countermeasure can be successfully developed at all, rather than ensuring that existing countermeasures can get fast regulatory approval or widespread distribution.

Personal protective equipment: for prioritizing survival, one might focus on developing entirely new types of advanced PPE, e.g. a self-sterilizing hazmat suit, rather than ensuring that we have N95s for everybody.

Geographic quarantine: for prioritizing survival, one might focus on 100% successful quarantine on an island like New Zealand or other naturally defensible areas, rather than partial quarantine improvements everywhere else in the world.

The general theme here is that rather than shoring up weak points (i.e. ‘trough defense’), survival is best ensured by making the strongest defense point even stronger (i.e. ‘peak defense’). The hypothesis is that if we were to succumb to a biological risk to the point of causing an existential catastrophe, it was because we entirely lacked some capability, rather than because we had some partial capability that wasn't widespread enough.[1]

I generally believe this hypothesis, although with a few big caveats. For example, even the strongest defense points might rely on widespread supply chains and thus depend on a fairly large/dispersed population that would need to be protected even with an emphasis on ‘peak defense’. For example, the Pfizer vaccine against COVID has something like 280 components from lots of different countries. If avoiding permanent collapse depended on even a small number of doses of that vaccine, getting to that point could still require total protection of many millions of people around the world.

In other words, building a handful of self-sterilizing hazmat suits is not going to cut it, and I expect many successful ‘peak defense’ strategies to still involve scaling to the point of protecting millions of people. But even with a more expansive version of peak defense, I expect that compared to normal pandemic preparedness measures, these strategies will look to many people like unhinged overkill (e.g. stockpiling 50 million super hazmat suits to protect every essential worker in the US, sequencing everything at all ports of entry for a small island, putting vaccine manufacturing capacity in a bunker, etc).

Another important point: some peak defense investments basically only matter for extreme worst cases (e.g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Countermeasures & substitution effects in biosecurity, published by ASB on December 16, 2021 on The Effective Altruism Forum.

An adversary who knows that his opponents’ troops have been inoculated against anthrax can switch his battle plans to smallpox or plague—or to an agent for which no vaccine exists. -Ken Alibek, Biohazard

A challenge for reducing bio risk is that many of the risks are coming from adversaries. Adversaries can react to our interventions, and so developing countermeasures may be less effective than one might naively expect due to 'substitution effects'.[1] There are several distinct substitution effects:

‘Switching’ - we find a countermeasure for X, adversary then switches from X to developing Y

‘Escalating’ - we find a countermeasure for X, adversary modifies X to X’ to overcome countermeasure[2]

‘Attention Hazard + Offense Bias’ - we investigate a countermeasure for X, but fail. Adversary was previously not developing X, but seeing our interest in X starts to develop X.

Can be combined with escalation, even if we successfully find countermeasure for X, adversary is now on general X pathway and starts developing X’

Can also simply be a timeframe effect, if adversary can produce X before we successfully get the countermeasure to X (although here it matters if the adversary is more like a terrorist and would deploy X as soon as it was created, or a state program that would keep X around for awhile, imposing some ongoing accident or warfare risk until the countermeasure for X was found).

Can be a problem if we think the countermeasure is imperfect enough that the attention hazard outweighs the countermeasure development.

‘Exposing Conserved Vulnerabilities’ - imagine that there are 10 possible attacks. For 9 of them, we could quickly develop a countermeasure in an emergency, but for one of them, finding a countermeasure is impossible (we don’t know which is which in advance). We research countermeasures for all 10 attacks, solve 9 of them, but also reveal the attack that is impossible to counter. The adversary thus picks that one, leaving us worse off than had we just remained ignorant and just waited for an emergency (e.g. if we assume before doing the research that adversary would have only a 10% chance of picking that attack). By picking up the low hanging fruit, we’ve ‘funneled’ our adversary towards the weak points.

Substitution effects will have varying consequences for global catastrophic biological risks (GCBRs). In a worst case scenario, finding countermeasures to more mundane things will cause adversaries to move towards GCBR territory (either by more heavily engineering mundane things, or switching into entirely new kinds of attack). However, this is also counterbalanced by the fact that bioweapons (‘BW’) in general might be less attractive when there are countermeasures for a lot of them.

‘Reduced BW appeal’ - An adversary has a program that is developing X and Y. We find a cure to X, which reduces the appeal of the entire program, causing the adversary to give up on both X and Y.

Better technology for attribution (e.g. tracing the origin of an attack or accident) is one concrete example that produces ‘reduced BW appeal.’ Better attribution is unlikely to dissuade development of bioweapons oriented towards mutually assured destruction (and we might expect most GCBRs to be coming from such weapons). But in reducing the strategic/tactical appeal of bioweapons for assassination, sabotage, or ambiguous attacks, the overall appeal of a BW program is reduced, which could have spillover effects into reducing the probability of more GCBR-style weapons.

One key question around substitution effects is the flexibility of an adversary. I get a vague impression from reading about the Soviet program that many scientists were extreme specialists, focus...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 12/16: On Your Marks, published by Zvi on December 16, 2021 on LessWrong.

The main action continues to be at the Omicron post. There’s still some background developments and near term developments worth noticing, but for now nothing that seems as important or that seems terribly urgent, so read that first.

Executive Summary

Omicron is here and will take over soon.

We might not do much about it until far too late.

And that’s probably better than the alternative.

Also some other stuff, like Will Eden’s excellent tweetstorm below.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 750k cases (+2%) and 8,800 deaths (+5%).

Results: 747k cases (+2%) and 8,813 deaths (+5%).

Prediction for next week: 862k cases (+15%) and 9,050 deaths (+3%).

Those were some pretty good predictions. Don’t count on it happening again any time soon. Stuff’s about to get weird.

The case growth depends on how far along we are in Omicron taking over, and when those tests results will come in and how many people will get tested if the cases are mild. If we take the 2.9% for the period ending 12/11 seriously, we now are moving ahead 12 days, I’ll guess the true general doubling time is 3 days, so for the sample we’ll get it’s going to be something like 18% (+12% from previous levels) Omicron, and cases of Delta likely remain roughly level, with more upside than downside for the count. It’s already at or last that level in the specific places we’re measuring, but it makes sense that they’d be ‘ahead’ of other areas with much larger populations. And we’ll likely see some mitigation. So I’ll go with +15%, but the error bars are wildly higher than before. So many variables.

Deaths

Deaths are straightforward, no reason to expect major change. Paxlovid remains illegal, Omicron isn’t here yet for long enough.

Cases

This is the question. We saw some regional movement last week, it’s probably largely random reporting fluctuations, except that the rise in the Northeast is very real. Going forward it’s all about Omicron. It’s super hard to time it, and what comes up on tests is several days behind what’s actually happening (e.g. a full doubling time) but my guess is that we’re already above 10% and increasing fast. London is our future, it’s merely unevenly distributed for a limited time.

So, how about Paxlovid? Can we hurry?

Paxlovid Remains Illegal

The final numbers are in, and they look good. Efficacy holding steady at 88%, no deaths in the treatment group, actively fewer ‘side effects’ in the treatment group compared to controls (presumably because controls had Covid-19, which causes a lot of ‘side effects’).

So now we have four questions:

When will Paxlovid become legal?

How much Paxlovid will be available?

While supply is limited, who gets that supply?

When supply isn’t limited, how do we get it to people in time?

The answer to when is around the end of the year. As of writing this the prediction market now again has it as a favorite at 55%, I hope some of you are making yourselves some money. There isn’t that much uncertainty here at this point, as I understand it, and we’ll almost certainly have Paxlovid by mid-January.

That brings us to question two. There are going to be enough doses this year for 200,000 patients. That’s two days worth of known cases right now, one day by year’s end, so indeed going forward production (as crippled by worries about approval) will be the central limiting factor. Pfizer will have 80 million treatments available next year, of which the US Government has bought 10 million. We don’t have information on the rate at which this can ramp up, which will be important because the bulk of the cases in 2022 are probably going to be in the first two months, and that’s also when we need to use Paxlovid to reduce pressure on hospitals.

Weekend Editor expe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reviews of “Is power-seeking AI an existential risk?”, published by Joe Carlsmith on December 16, 2021 on LessWrong.

Open Philanthropy solicited reviews of my draft report “Is power-seeking AI an existential risk?” from various sources. Where the reviewers allowed us to make their comments public in this format, links to these comments are below, along with some responses from me in blue.

Leopold Aschenbrenner

Ben Garfinkel

Daniel Kokotajlo

Neel Nanda

Nate Soares

Christian Tarsney

David Thorstad

David Wallace

Anonymous 1 (software engineer at AI research team)

Anonymous 2 (academic computer scientist)

The table below (spreadsheet link here) summarizes each reviewer’s probabilities and key objections.

An academic economist focused on AI also provided a review, but they declined to make it public in this format.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Biosecurity needs engineers and materials scientists, published by willbradshaw on December 16, 2021 on The Effective Altruism Forum.

TL;DR: Expertise in engineering physical systems is critically needed for some of the most important and neglected biosecurity interventions, including improved PPE and designing pandemic-safe buildings. If you have that expertise and are interested in getting involved, you should contact me.

Disclaimer: All uses of "engineering" in this post refer to the engineering of physical systems (e.g. materials engineering or civil engineering). Sorry, software engineers.

The problem

Historically, the EA-adjacent biosecurity space has been dominated by people with backgrounds in biology, epidemiology, medicine, and public policy. Most of the junior people thinking about working on biosecurity also have these backgrounds. These seem to be the people who the EA community thinks of as having something to contribute to biosecurity.

This makes sense: the creation of biological threats is a political & social problem, and the threats themselves are, well, biological, so people with expertise in biology & medicine seem well-equipped to think about countermeasures. All of these areas of expertise will be needed to tackle the biosecurity threat.

However, many of the most important interventions for reducing catastrophic biological risk are neither biological nor social in nature. Instead, these interventions use physical means to block, capture or destroy pathogens. Such technologies are much more threat-agnostic than most interventions that rely on biotechnology, providing broad protection while necessitating far less dual-use research and infohazardous threat modelling.

Rather than biologists and policymakers, the people best-equipped to drive these interventions forward are materials scientists, building engineers, and others with strong backgrounds in the applied physical sciences. Unfortunately, people with these backgrounds are currently severely lacking in biosecurity.

The interventions

There are many ways people with applied physical science expertise could have an impact in biosecurity. I've outlined a few of the most important and/or salient for me below; I'm sure there are others I'm not yet thinking of.

(For more on these and other exciting technological interventions, read Appendix A of The Apollo Program for Biodefense.)

Physical protection against pathogens

As mentioned above, the most important biosecurity interventions engineers could work on are those intended to stop disease causing agents getting into people in the first place. In roughly descending order of importance, these include:

Improving personal protective equipment (PPE): The design of face masks and other PPE has barely changed for many decades, and still receives relatively little attention. PPE that was highly effective, easy to use, and cheap to distribute would be perhaps the single most transformative technological intervention to overcome biological risks, but is currently laughably neglected.

Suppressing pathogen spread in the built environment: Indoor spaces are far more dangerous than outdoor spaces for transmitting respiratory infections. While not as general or powerful as improved PPE, interventions to reduce transmission in building could significantly slow the spread of severe biological threats. Proposed interventions in this area include improvements in ventilation, far-UVC irradiation, and upper-room UVGI. However, many of these have significant difficulties to overcome before widespread adoption, and relatively little time and money have gone into either implementing these technologies or identifying promising alternatives.

Improving biosafety in high-containment labs and clinics: As with PPE for general use, the technologies and systems used to maintain b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Where can one learn deep intuitions about information theory?, published by Valentine on December 16, 2021 on LessWrong.

I'm currently going through Brilliant's course on "Knowledge and Uncertainty". I just got through the part where it explains what Shannon entropy is. I'm now watching a wave of realizations cascade in my mind. For instance, I now strongly suspect that the "deep law" I've been intuiting for years that makes evolution, economics, and thermodynamics somehow instances of the same thing is actually an application of information theory.

(I'm honestly kind of amazed I was able to follow as much of rationalist thought and Eliezer's writings as I was without any clue what the formal definition of information was. It looks to me like it's more central than is Bayes' Theorem, and that it provides essential context for why and how that theorem is relevant for rationality.)

I'm ravenous to grok more. Sadly, though, I'm bumping into a familiar wall I've seen in basically all other technical subjects: There's something of a desert of obvious resources between "Here's an article offering a quick introduction to the general idea using some fuzzy metaphors" and "Here's a textbook that gives the formal definitions and proofs."

For instance, the book "Thinking Physics" by Lewis Carroll Epstein massively helps to fill this gap for classical physics, especially classical mechanics. By way of contrast, most intro to physics textbooks are awful at this. ("Here we derive the kinematic equation for an object's movement under uniform acceleration. Now calculate how far this object goes when thrown at this angle at this velocity." Why? Is this really a pathway optimized for helping me grok how the physical world works? No? So why are you asking me to do this? Oh, because it's easy to measure whether students get those answers right? Thank you, Goodhart.)

Another excellent non-example is the Wikipedia article on how entropy in thermodynamics is a special case of Shannon entropy. Its length is great as a kind of quick overview, but it's too short to really develop intuitions. And it also leans too heavily on formalism instead of lived experience.

(For instance, it references shannons (= bits of information), but it gives no hint that what a shannon is measuring is the average number of yes/no questions of probability 1/2 that you have to ask to remove your uncertainty. Knowing that's what a shannon is (courtesy of Brilliant's course) gives me some hint about what a hartley (= base ten version instead of base two) probably is: I'm guessing it's the average number of questions with ten possible answers each, where the prior on each answer is 1/10, that you'd have to ask to remove your uncertainty. But then what's a nat (= base e version)? What does it mean for a question to have an irrational number of possible equally likely answers? I'm guessing you'd have to take a limit of some kind to make sense of this, but it's not immediately obvious to me what that limit is let alone how to intuitively interpret what it's saying. The Wikipedia article doesn't even hint at this question let alone start to answer it.

It's quite happy just to show that the algebra works out.)

I want to learn to see information theory in my lived experience. I'm fine with technical details, but I want them tied to intuitions. I want to grok this. I don't care about being able to calculate detailed probabilities or whatever except inasmuch as my doing those exercises actually helps with grokking this.

Even a good intuitive explanation of thermodynamics as seen through the lens of information theory would be helpful.

Any suggestions?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evidence Sets: Towards Inductive-Biases based Analysis of Prosaic AGI, published by Ameya Prabhu on December 16, 2021 on The AI Alignment Forum.

Epistemic Status: Exploratory. My current but-changing outlook with limited exploration & understanding for ~60-80hrs. Hence, most provocative-seeming claims should be mistakes, except claims in all ‘Application’ sections.

Acknowledgements: This post was written under Evan Hubinger’s direct guidance and mentorship as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Thanks to particlemania, Shashwat Goel and Shawn Hu for exciting discussions. They might not agree with some of the claims made here; all mistakes are mine.

Summary (TL;DR)

Goal: Understanding the inductive biases of Prosaic AI systems could be very informative towards creating a frame of safety problems and solutions. The proposal here is to generate an Evidence Set from current ML literature to model the potential inductive bias of Prosaic AGI.

Procedure: In this work, I collect evidence of inductive biases of deep networks by studying ML literature. Moreover, I estimate from current evidence whether these inductive biases vary with scaling to large models. If a phenomenon seems robust to or amplified by scaling, I discuss it here and add it to the Evidence Set.

Structure: I provide interpretations of some interesting papers to AI safety in three maximally-relevant subareas of ML literature (pretraining-> finetuning, generalization and adversarial robustness), and demonstrate use cases to AI safety (in ‘Application’ subsections). I then summarize evidence from each area to form the Evidence Set.

Current Evidence Set: Given in Section “Evidence Set” (last section of this post)

Inspiration: I think developing good intuitions about inductive biases and past evidence essentially constitutes ‘experience’ used by ML researchers. Evidence sets and broadly analyzing inductive biases are the first steps towards mechanizing this intuition. Inductive bias analysis might be the right level of abstraction. A large, interconnected Evidence Set might give a degree of gears-level understanding of the black-box that is deep networks.

Applications: Any theory of inductive biases, if fully developed, can sufficiently explain the observations collected in the evidence set. Alternatively, we can use the evidence set to falsify mechanistic statements about training rationales in Training stories. However, I expect evidence sets to be helpful beyond these two applications.

Limitations: Evidence sets, and more broadly, inductive bias analysis, seems useful for tackling inner-alignment and not aligned objective-learning (outer-alignment). Secondly, the predictive power of evidence sets is unclear. When the proposal is fully fleshed out, it could have lower predictive power on properties or possible outcomes of training rationales, limiting its applicability. Thirdly, it’s challenging to mechanize or automate & scale inductive bias analysis, unlike say, transparency tools. Nevertheless, I think the idea of evidence sets is a promising proposal.

A: Where does it Fit in Alignment Literature?

Situating in Safety Scenarios: Studying inductive biases of Prosaic AI seems well-suited for systems arising from the scaling hypothesis. More broadly, from Dafoe’s AGI perspectives, studying inductive biases could be instrumental for safety considerations in the case of General Purpose Technology and may be helpful for the AI Ecology system classes. A-priori thinking about inductive biases of super-intelligent agents looks very unintuitive to me as I cannot currently understand how ML literature could meaningfully inform inductive biases of intelligent agents which could tweak themselves[1].

Direct Applications: I list two applications where our evidence ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #7, published by Zvi on December 16, 2021 on LessWrong.

Here are some better worlds we might have lived in, but don’t.

In a better world, I could focus on this full time and also maybe even hire a research assistant, and be better able to scour for information.

In a better world than that, there would be a department at a newspaper, or some other such place, that scoured the world for data sources like SGTF deletion counts and wastewater concentrations that could help us understand spread, called the sources to attempt to get the best data possible, and complied it all in nice form.

In an even better world than that, there would be funding, either public or private, for the actual gathering of additional data.

In an even better world than that, we’d have run challenge trials the moment Omicron was detected, and we’d already have all our answers.

Alas, instead, we live in this world, where none of those things are true. I am grateful to all my news sources and those who help gather the information together, and everyone working tirelessly in the lab, but the whole thing is still a slapstick hodgepodge, and we know so much less than we could know so much slower than we could know it.

Still, we do the best we can.

This time out, we have a few new places whose data has been located, to supplement the United Kingdom and Denmark. We also have several important new lab results.

There is some uncertainty about how far along things are. There is uncertainty regarding how much milder Omicron is than Delta, from it being essentially the same (but much better on a per-case basis due to who it infects) to there being a substantial or even large difference.

But the main uncertainty that I’m now wondering about, that feels central, is how we will react.

What will happen when the rice grains on the chessboard suddenly get fully out of hand, stuff hits the fan and the hospitals overflow? Not if. When. How will governments react? How will the people react?

Last year, I created a toy spreadsheet to model how the rise of Alpha might go, and whether the control system (governments and individuals reacting to the levels of cases, hospitalizations and deaths by scaling up or down prevention measures) combined with vaccinations could handle it. I invite everyone to copy, modify, mess around and find out, and create their own toy models. There are a bunch of things in column AH that you can modify to see what they imply. Note that this sheet is designed for my own use rather than public consumption; it seemed useful to share it anyway but it’s very much not optimized for that.

Prediction markets believe and I even more strongly believe that Omicron will become the majority of cases by year’s end in the USA and most of Europe. The question is what happens after that. Will we be able to make the straight line on a logarithmic graph bend before things get completely out of hand, one more time? Or will this be the time it’s too much and we fail?

What are we willing to do to try and not fail? What are we willing to do to be able to claim that we were trying not to fail, even if failure is inevitable? These are the questions.

Meanwhile, there’s still the question of exactly where we are and what are the physical rules of the game, so let’s review the evidence.

United Kingdom

Well, that escalated quickly.

50% Omicron yesterday in London, 60% today, stunning growth rate, the daily cases chart’s last three days are still incomplete.

Other regions too, but not as dramatically, here’s the data source:

A day older but better labeled:

And finally another illustration for London from one day ago:

And the synthesis for the nation.

It’s a strange dynamic where Delta is stable in London and stable outside of London, yet in London Omicron goes vertical much faster than elsewhere. It’s a big hint ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High School Seniors React to 80k Advice, published by johnburidan on December 16, 2021 on The Effective Altruism Forum.

"That article made me angry. All this time was spent on making sure your job was helping others and no time was spent on whether it brought you joy." I assigned high school seniors in our economics class to read Benjamin Todd's summary essay on planning a high-impact career. Some of the students were offended by the essay. Here's how it went.

In general, I teach career advice using the What Color is Your Parachute? framework. I really like the framework, because it provides a lot of direction for thinking about one's own desires and how those can be corralled to serve to a higher mission. This year I added two segments to the career advice, one on globalization, and another on EA. This is in the context of an Econ class, remember, so I felt justified this year in leaning in on these frameworks.

MRU has a nice little segment about how to think about a career given globalization. We started with international trade and made our way to a discussion on globalization and the elephant graph and what that means for careers. I thought this was depressing, because of the narrow number of careers in which one can expect to see wage growth. So we also discussed how much easier it is to be the best person with a set of three skills than the best person with one skill, as an example of how one can still be competitive amidst global competition. That combinatorial insight is very helpful.

Next, there is much more to a career than maximizing profit in the context of globalization. And I would be surprised if more than 10% of that class end up in jobs with the term 'engineer' attached, so we talked about other high-impact careers. I gave them this Benjamin Todd essay, and they came to class with a ton of pushback.

In the first ten minutes of class, before I even had said a word, they laid out a series of arguments against the EA approach to career advice. I relished this and took notes as they went. Such good material, such familiar objections! These very normal, American students from median American households in a non-coastal city sensed something deeply challenging about EA, but these are challenges which I think the community has already addressed.

Objection 1: Giving you life to others in this obsessive way will result in burnout.

Objection 2: This type of advice is for "people-pleasers", not independent, self-sufficient people.

Objection 3: This type of advice is for only super-self-sufficient people who want to take on huge responsibilities. I might not want big responsibilities.

Objection 4: This type of advice ignores the indirect value of normal careers, like working in shipping logistics.

Objection 5: If everyone tried to have a high-impact career, then the career wouldn't be high-impact anymore.

Against these objections is the material in the essay itself. Close reading is hard! "You can divide career aims into three categories: (i) personal priorities (ii) impartial positive impact, and (iii) other moral values." And:

Turning to personal priorities, research suggests that people are most satisfied when they have work that’s:

Meaningful

Something they’re good at

Engaging & with autonomy"

Properly understood, these two quotes answer Objections 1, 2, and 3. Burnout is antithetical to what should be a personal priority: personal flourishing with a healthy mind and healthy body. But I found it fascinating that several students, having no previous exposure to EA, immediately worried about burnout. I wonder if the lack of vocabulary about individual psychology sounds an internal alarm for postulants? Secondly, EA is about finding the right fit, and that does mean knowing your own strengths, weaknesses, and desire for responsibility.

A close reading together ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Elicitation for Modeling Transformative AI Risks, published by David Manheim on December 16, 2021 on The AI Alignment Forum.

This post is part 8 in our sequence on Modeling Transformative AI Risk. We are building a model to understand debates around existential risks from advanced AI. The model is made with Analytica software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final output corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the Introduction post. Unlike other posts in the sequence, this discusses the related but distinct work around Elicitation.

We are interested in feedback on this post, but to a greater extent than the other posts, we are interested in discussing what might be useful, and how to proceed with this. We would also welcome discussion from people working independently on elicitations, as we have discussed this extensively with other groups, many of whom are doing related work.

As discussed in previous posts in this series, the model we have built is a tentative one, and requires expert feedback and input. The traditional academic method for getting such feedback and input is usually referred to as elicitation, and an extensive field of academic work discusses how this can best be done. (As simple examples, this might include eliciting an estimated cost, and probability distribution, or a rank ordering of which outcomes from a project are most important.)

Elicitation of expert views is particularly critical in AI safety for both understanding debates between experts and representing the associated probabilities. At the same time, many elicitation and forecasting projects have the advantage of unambiguous and concrete questions with answers that will be observed in the near term, or ask for preferences about outcomes which are well understood. Because these advantages are mostly absent for AI safety questions, the focus in this project is on understanding debates (instead of attempting to settle debates that are already understood, or are not resolvable even in theory). This means that there is no intent to elicit a “correct” answer to questions which may be based on debated or disputed assumptions. For this reason, we have taken an approach designed to start with better understanding experts’ views of the domain overall, rather than focus on the outcomes directly.

This leads to opportunities for better understanding the sources of disagreement.

The remainder of this post first discusses what elicitation can and should be able to accomplish in this domain, and for this project, as well as what conceptual and actual approaches we are using. This should help explain how elicited information can inform the concrete model, which then can then hopefully help inform decisions - or at least clarify why the decisions about approaches to take are disputed. Following that, we outline our tentative future plan, and what additional steps for elicitation may look like.

What to do about forecasting given uncertainties and debates?

In domains where the structure of uncertainties are clear, and not debated, it is possible to build a model similar to that built in the current project, ask experts whether the structure is correct, and based on their input, build a final Directed Acyclic Graph or other representation of the joint distribution that correctly represents their views. After this, we would ask experts to attach probability distributions to the various uncertainties, perhaps averaging their opinions for each node in the DAG, so that we could get quantitative predictions for the outcomes via Monte Carlo.

Long-term forecasting of deeply uncertain and debated outcomes ...