The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Scott on Abolishing the FDA, published by Maxwell Tabarrok on December 15, 2023 on LessWrong.

Scott Alexander's

recent post on the FDA raises the average level of discourse on the subject. He starts from the premise that the FDA deserves destruction but cautions against rash action. No political slogan can be implemented without clarification and "Abolish the FDA" is no different, but Scott's objections aren't strong reasons to stop short of a policy implementation that still retains the spirit behind the slogan.

Scott's preferred proposal is to essentially keep the authority and structure of the FDA the same but expand the definition of supplements and experimental drugs. This way, fewer drugs are illegal but there aren't big ripple effects on the prescription and health insurance systems that we have to worry about.

The more hardline libertarian proposal is to restrict the FDA's mandatory authority to labeling and make their efficacy testing completely non-binding. This would turn the FDA into a informational consumer protection agency rather than a drug regulator. They can slap big red labels on non-FDA approved drugs and invite companies to run efficacy tests to get nice green labels instead, but they can't prevent anyone from taking a drug if they want it.

Let's go through Scott's objections to the hardline plan and see if they give good reasons to favor one over the other.

Are we also eliminating the concept of prescription medication?

I can see some "If I were king of the world" overhauls to the health system that might do away with mandatory prescriptions, but I think the point of this exercise is to see if we can abolish the FDA without changing anything else and still come out ahead, accounting for costly second-order effects from the rest of the messed up health system. So no, the hardline "abolish the FDA" plan would not remove the legal barrier of prescription. Here is Scott's response:

But if we don't eliminate prescriptions, how do you protect prescribers from liability? Even the best medications sometimes cause catastrophic side effects. Right now your doctor doesn't worry you'll sue them, because "the medication was FDA-approved" is a strong defense against liability. But if there are thousands of medications out there, from miraculous panaceas to bleach-mixed-with-snake-venom, then it becomes your doctor's responsibility to decide which are safe-and-effective vs.

dangerous-and-useless. And rather than take that responsibility and get sued, your doctor will prefer to play it safe and only use medications that everyone else uses, or that were used before the FDA was abolished.

This is a reasonable concern, litigation pressure is a common culprit behind

spiraling regulatory burden. But in this case we can be confident that turning the FDA into a non-binding informational board won't turn prescriptions into an even higher legal hurdle because doctors already prescribe far outside of FDA approval.

When a drug is tested by the FDA it is tested as a treatment for a specific condition, like diabetes or throat cancer. If the drug is approved, it is approved only for the outcome measured in efficacy testing and nothing else. However, doctors know that certain drugs approved for one thing are effective at treating others. So they can issue an "off-label" prescription based on their professional opinion.

Perhaps 20% of all prescriptions in the US are made off-label and more than half of doctors make some off label prescriptions.

So doctors are clearly willing to leave the legal umbrella of FDA approval when they make prescription decisions. There are lots of high profile legal cases about off-label prescriptions but they are mostly about marketing and they haven't dampened doctor's participation in the practice. If doctors were comfortable enough to

pres...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My quick thoughts on donating to EA Funds' Global Health and Development Fund and what it should do, published by Vasco Grilo on December 15, 2023 on The Effective Altruism Forum.

I think there is a strong case for donating to EA Funds'

Global Health and Development Fund (GHDF) if one wants to support interventions in

global health and development without attending to their

effects on animals. On the other hand, given this goal, I believe one had better donate to GiveWell's

All Grants Fund (AGF) or

unrestricted funds (GWUF), or Giving What We Can's (GWWC's)

Global Health and Wellbeing Fund (GHWF). In addition, I encourage GHDF to:

Let its donors know that donating to GHDF in its current form has a similar effect to donating to AGF (if that is in fact the case).

Consider appointing additional fund managers independent from GiveWell.

Consider accepting applications.

In any case, the goal of this post is mostly about starting a discussion about the future of GHDF rather than providing super informed takes about it. So feel free to share your thoughts or vision below!

Case for donating to GiveWell's All Grants Fund or unrestricted funds

Donating to AGF or GWUF instead of GHDF seems better if one highly trusts GiveWell's prioritisation:

Donating to GHDF in its current form appears to have the same effect as donating to AGF or GWUF:

Like AGF and GWUF, GHDF "aims to improve the health or economic empowerment of people around the world as effectively as possible".

My understanding is that GHDF makes more uncertain or riskier grants than GiveWell's

Top Charities Fund[1] (TCF), but AGF,

launched in August 2022, now makes such grants too. AGF

funds:

GiveWell's top charities.

Organisations implementing potentially cost-effective and scalable programs.

Established organisations implementing cost-effective programs that GiveWell does not expect to scale.

Organisations aiming to influence public health policy.

Organisations producing research to aid our grantmaking process.

Organizations that raise funds for our recommended charities.

GHDF "is managed by Elie Hassenfeld,

GiveWell's co-founder [and CEO]".

GHDF does not accept applications, and neither does AGF.

People in the United Kingdom can support GiveWell's funds and top charities through tax deductible donations via

GiveWell UK, which

was launched in August 2022 as AGF.

Having EA Funds as an additional intermediary seems unnecessary unless it is doing some extra evaluation, which does not appear to be the case.

As a side note, I would also say there is a pretty small difference between which one of GiveWell's

funds, TCF, AGF or GWUF, one donates to:

Due to

funging, more donations to TCF will result in AGF granting less money to GiveWell's

top charities.

GiveWell arguably has tiny room for more funding given Open Philanthropy's

support, so donating to GWUF is similar to donating to AGF[2].

However, if you highly trust GiveWell's prioritisation, donating to GWUF

is the best option given its greatest flexibility, followed by the AGF and TCF. Yet, donors may prefer donating to TCF to facilitate explanations of their effective giving (e.g. skipping the need to go into expected value or

funging).

Case for donating to Giving What We Can's Global Health and Wellbeing Fund

Donating to GHWF instead of GHDF seems better if one:

Welcomes further evaluation of the process behind the recommendations of GiveWell and other evaluators in the global health and wellbeing space (e.g. Happier Lives Institute), trusts GWWC's research team to identify evaluators to rely on, and wants the evaluations to be published, as in GWWC's

evaluations of evaluators. These would be my main reasons for donating to GHWF instead of GHDF, which has not produced public evaluations of GiveWell's recommendations.

Is open to donating to funds or organisations not suppo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Current AIs Provide Nearly No Data Relevant to AGI Alignment, published by Thane Ruthenis on December 15, 2023 on The AI Alignment Forum.

Recently, there's been a fair amount of pushback on the "canonical" views towards the difficulty of AGI Alignment (the views I call the "least forgiving" take).

Said pushback is based on empirical studies of how the most powerful AIs at our disposal currently work, and is supported by fairly convincing theoretical basis of its own. By comparison, the "canonical" takes are almost purely theoretical.

At a glance, not updating away from them in the face of ground-truth empirical evidence is a failure of rationality: entrenched beliefs fortified by rationalizations.

I believe this is invalid, and that the two views are much more compatible than might seem. I think the issue lies in the mismatch between their subject matters.

It's clearer if you taboo the word "AI":

The "canonical" views are concerned with scarily powerful artificial agents: with systems that are human-like in their ability to model the world and take consequentialist actions in it, but inhuman in their processing power and in their value systems.

The novel views are concerned with the systems generated by any process broadly encompassed by the current ML training paradigm.

It is not at all obvious that they're one and the same. Indeed, I would say that to claim that the two classes of systems overlap is to make a very strong statement regarding how cognition and intelligence work. A statement we do not have much empirical evidence on, but which often gets unknowingly, implicitly snuck-in when people extrapolate findings from LLM studies to superintelligences.

It's an easy mistake to make: both things are called "AI", after all. But you wouldn't study manually-written FPS bots circa 2000s, or MNIST-classifier CNNs circa 2010s, and claim that your findings generalize to how LLMs circa 2020s work. By the same token, LLM findings do not necessarily generalize to AGI.

What the Fuss Is All About

To start off, let's consider where all the concerns about the AGI Omnicide Risk came from in the first place.

Consider humans. Some facts:

Humans posses an outstanding ability to steer the world towards their goals, and that ability grows sharply with their "intelligence". Sure, there are specific talents, and "idiot savants". But broadly, there does seem to be a single variable that mediates a human's competence in all domains. An IQ 140 human would dramatically outperform an IQ 90 human at basically any cognitive task, and crucially, be much better at achieving their real-life goals.

Humans have the ability to plot against and deceive others. That ability grows fast with their g-factor. A brilliant social manipulator can quickly maneuver their way into having power over millions of people, out-plotting and dispatching even those that are actively trying to stop them or compete with them.

Human values are complex and fragile, and the process of moral philosophy is more complex still. Humans often arrive at weird conclusions that don't neatly correspond to their innate instincts or basic values. Intricate moral frameworks, weird bullet-biting philosophies, and even essentially-arbitrary ideologies like cults.

And when people with different values interact...

People who differ in their values even just a bit are often vicious, bitter enemies. Consider the history of heresies, or of long-standing political rifts between factions that are essentially indistinguishable from the outside.

People whose cultures evolved in mutual isolation often don't even view each other as human. Consider the history of xenophobia, colonization, culture shocks.

So, we have an existence proof of systems able to powerfully steer the world towards their goals. Some of these system can be strictly more powerfu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "AI Alignment" is a Dangerously Overloaded Term, published by Roko on December 15, 2023 on LessWrong.

Alignment as Aimability or as Goalcraft?

The Less Wrong and AI risk communities have obviously had a huge role in mainstreaming the concept of risks from artificial intelligence, but we have a serious terminology problem.

The term "AI Alignment" has become popular, but people cannot agree whether it means something like making "Good" AI or whether it means something like making "Aimable" AI. We can define the terms as follows:

AI Aimability = Create AI systems that will do what the creator/developer/owner/user intends them to do, whether or not that thing is good or bad

AI Goalcraft = Create goals for AI systems that we ultimately think lead to the best outcomes

Aimability is a relatively well-defined technical problem and in practice almost all of the technical work on AI Alignment is actually work on AI Aimability. Less Wrong has for a long time been concerned with Aimability failures (what Yudkowsky in the early days would have called "Technical Failures of Friendly AI") rather than failures of Goalcraft (old-school MIRI terminology would be "Friendliness Content").

The problem is that as the term "AI Alignment" has gained popularity, people have started to completely merge the definitions of Aimability and Goalcraft under the term "Alignment". I recently ran some Twitter polls on this subject, and it seems that people are relatively evenly split between the two definitions.

This is a relatively bad state of affairs. We should not have the fate of the universe partially determined by how people interpret an ambiguous word.

In particular, the way we are using the term AI Alignment right now means that it's hard to solve the AI Goalcraft problem and easy to solve the Aimability problem, because there is a part of AI that is distinct from Aimability which the current terminology doesn't have a word for.

Not having a word for what goals to give the most powerful AI system in the universe is certainly a problem, and it means that everyone will be attracted to the easier Aimability research where one can quickly get stuck in and show a concrete improvement on a metric and publish a paper.

Why doesn't the Less Wrong / AI risk community have good terminology for the right hand side of the diagram? Well, this (I think) goes back to a decision by Eliezer from the SL4 mailing list days that one should not discuss what the world would be like after the singularity, because a lot of time would be wasted arguing about politics, instead of the then more urgent problem of solving the AI Aimability problem (which was then called the control problem).

At the time this decision was probably correct, but times have changed. There are now quite a few people working on Aimability, and far more are surely to come, and it also seems quite likely (though not certain) that Eliezer was wrong about how hard Aimability/Control actually is.

Words Have Consequences

This decision to not talk about AI goals or content might eventually result in some unscrupulous actors getting to define the actual content and goals of superintelligence, cutting the X-risk and LW community out of the only part of the AI saga that actually matters in the end.

For example, the recent popularity of the e/acc movement has been associated with the Landian strain of AI goal content - acceleration towards a deliberate and final extermination of humanity, in order to appease the Thermodynamic God. And the field that calls itself AI Ethics has been tainted with extremist far-left ideology around DIE (Diversity, Inclusion and Equity) that is perhaps even more frightening than the Landian Accelerationist strain. By not having mainstream terminology for AI goals and content, we may cede the future of the universe to extremis...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Surveys on Community Health, Causes, and Harassment, published by David Moss on December 15, 2023 on The Effective Altruism Forum.

We are announcing a supplementary survey to gather timely information from the EA community before the next EA Survey in 2024.

This survey will contain questions related to:

Community health and satisfaction with the EA community

Cause prioritization and how EA resources should be allocated

Demographics (which can optionally be skipped if you provided your email address last time and opt for us to link your responses)

We are also sending out a separate survey, requested by CEA's Community Health and Special Projects team, focusing primarily on sexual harassment and gender-related experiences:

4. EA Climate and Harassment Survey

You can take the first survey here. This will give you the option to take the Climate and Harassment Survey immediately afterwards, without having to answer the demographic questions twice.

Alternatively, you can just take the Climate and Harassment survey here.

If you wish to share links to either of these surveys with others, please use the following links:

Both surveys:

https://rethinkpriorities.qualtrics.com/jfe/form/SV_1G37guBPVAl9TtI?source=sharing

Climate and Harassment Survey alone:

https://rethinkpriorities.qualtrics.com/jfe/form/SV_bxD0wtmuuXw4KUe?source=sharing

The first survey should be significantly shorter than the main EA Survey, depending on how much detail you choose to provide in the open comment questions and whether you skip the demographic section by providing your email address. The EA Climate and Harassment Survey is estimated to take between 5 and 30 minutes depending on how much detail you choose to provide.

Both surveys are planned to close on 1st January 2024.

Acknowledgements

The post is a project of Rethink Priorities, a global priority think-and-do tank, aiming to do good at scale. We research and implement pressing opportunities to make the world better. We act upon these opportunities by developing and implementing strategies, projects, and solutions to key issues. We do this work in close partnership with foundations and impact-focused non-profits or other entities. If you're interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU policymakers reach an agreement on the AI Act, published by tlevin on December 15, 2023 on LessWrong.

On December 8, EU policymakers

announced an agreement on the AI Act. This post aims to briefly explain the context and implications for the governance of global catastrophic risks from advanced AI. My portfolio on Open Philanthropy's AI Governance and Policy Team includes EU matters (among other jurisdictions), but I am not an expert on EU policy or politics and could be getting some things in this post wrong, so please feel free to correct it or add more context or opinions in the comments!

If you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via

this short Google form.

Context

The AI Act has been in the works since 2018, and for the last ~8 months, it has been in the "trilogue" stage. The EU Commission, which is roughly analogous to the executive branch (White House or 10 Downing Street), drafted the bill; then, the European Parliament (analogous to the U.S. House of Representatives, with population-proportional membership from each country) and the Council of the EU (analogous to the U.S.

conference committees in the US Congress).

In my understanding, AI policy folks who are worried about catastrophic risk were hoping that the Act would include regulations on all sufficiently capable GPAI (general-purpose AI) systems, with no exemptions for open-source models (at least for the most important regulations from a safety perspective), and ideally additional restrictions on "very capable foundation models" (those above a certain compute threshold), an idea floated by some negotiators in October.

threat assessments/dangerous capabilities evaluations and

cybersecurity measures, with a lot of the details to be figured out later by that Office and by standard-setting bodies like

CEN-CENELEC's JTC-21.

GPAI regulations

appeared in danger of being excluded after Mistral, Aleph Alpha, and the national governments of France, Germany, and Italy objected to what they perceived as regulatory overreach and threatened to derail the Act in November. There was also some reporting that the Act would totally exempt open-source models from regulation.

What's in it?

Sabrina Küspert, an AI policy expert working at the EU Commission, summarized the results on some of these questions in a

thread on X:

The agreement does indeed include regulations on "general-purpose AI," or GPAI.

There does appear to be a version of the "very capable foundation models" idea in the form of "GPAI models with systemic risks," which are based on capabilities and "reach," which I think means how widely deployed they are.

It looks like GPAI models are

presumed to have these capabilities if they're trained on 10^25 FLOP, which is one order of magnitude smaller than the October 30 Biden executive order's cutoff for reporting requirements (and which would

probably include GPT-4 and

maybe Gemini, but

no other current models as far as I know).

Küspert also says "no exemptions," which I interpret to mean "no exemptions to the systemic-risk rules for open-source systems."

Other reporting suggests there are wide exemptions for open-source models, but the requirements kick back in if the models pose systemic risks. However, Yann LeCun is

celebrating based on this part of a

Washington Post article: "The legislation ultimately included restrictions for foundation models but gave broad exemptions to "open-source models," which are developed using code that's freely available for developers to alter for their own products and tools. The move could benefit open-source AI companies in Europe that lobbied against the law, including France's Mistral and Germany's Aleph Alpha, as well as Meta, which relea...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some for-profit AI alignment org ideas, published by Eric Ho on December 14, 2023 on LessWrong.

Summary

This is a brain dump of some for-profit AI alignment organization ideas, along with context for why I believe a for-profit alignment organization can make a big contribution to AI safety. This is far from a complete list, and I welcome ideas and feedback. Also, if anyone wants to or is working on any of these ideas, I'd be happy to support in any way I can!

Context

I'm Eric, formerly co-founder of RippleMatch, an AI recruiting company with ~$80M raised, millions of users, and ~10% of the Fortune 500 as customers. I made the difficult decision to leave RippleMatch this year because I'm concerned about catastrophic risk from AI, and have been spending the last year thinking about ways to help. Given my background, I've been thinking a lot about for-profit ideas to help with alignment - many that can be VC-backed. Some of these ideas speak more directly to reducing catastrophic risk than others, but I think that all can put a founder in a strong position to help in the future.

Why I believe for-profit alignment orgs are valuable

I don't think for-profit approaches are inherently better than building non-profits, pursuing government regulation, or other approaches, but I think that for-profit orgs can make a substantial impact while attracting a different pool of talent eager to work on the problem.

With VC dollars, a for-profit organization can potentially scale far more quickly than a non-profit. It could make a huge impact and not have its growth capped by donor generosity. As a result, there can be far more organizations working on safety in the ecosystem tapping into a different pool of resources. That said, any VC-backed company has a relatively low chance of success, so it's a riskier approach.

Fundamentally, I believe that risk and compliance spend will grow extremely quickly over the coming decade, scaling with generative AI revenue. With comps in finance and cybersecurity, I'd guess that mid to high single digit percentages of overall AI spend will be on risk and compliance, which would suggest big businesses can be built here. Many startups tackling alignment will need to start by addressing short term safety concerns, but in doing so will position themselves to tackle long-term risks over time.

Onto the actual ideas!

Robustness approaches

Testing / benchmarking software

Test case management needs to look very different for LLMs compared to typical software. The idea is to sell companies deploying LLMs a SaaS platform with the ability to generate and manage test cases for their LLMs to make sure they are performing properly and ensure that performance doesn't drift from version to version. This startup would also incorporate a marketplace of common benchmarks that companies can pull off the shelf if relevant to their use case (e.g. common adversarial prompts).

Currently, my impression is that most companies don't use any software to manage their language model test suites, which is a problem given how often an LLM can fail to produce a good result.

Red-teaming as a service

Just as software companies penetration test their software, companies that use LLMs as well as companies who build frontier models will need to red-team their models with a wide variety of adversarial prompts. This would mostly test models for how they handle misuse and make them more robust against jailbreaking.

Just as a proper penetration test employs both manual and automated penetration testing, this startup would require building / fine-tuning the best automated red-teaming LLM that likely draws on multiple frontier models, as well as employ the best manual red-teamers in the space. Enterprises would likely pay a subscription depending on their usage, which would likely be spiky.

The...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Love, Reverence, and Life, published by Elizabeth on December 14, 2023 on LessWrong.

Vegan advocates frequently argue that compromise positions like heavily reduced but nonzero meat consumption, humane certifications, or choosing meat with a lower suffering profile are not acceptable. The usual reason given is that the compromises aren't emotionally sustainable, and people inevitably slide back into full blown omnivorism. I (Elizabeth) never found this satisfying, emotionally or logically, and follow up discussions never went anywhere useful. Recently* Tristan gave an answer I did find satisfying, and made me suspect a follow-up discussion would be highly educational.

This is that follow up discussion, and it was indeed very educational. We dove deep into what taking reverence for life as your central value might mean, and how failing to center on this might be risky or invite some degree of sterility. I (Tristan) felt able to express some views I'm not always able to convey, and deeply appreciated the continued curiosity and help forgining those views that occured throughout. And though we might still hold quite differing views at the end of the day, this feels like a further step taken in epistemic good will that will hopefully help foster more conversations like it in the future.

*Well, it was recent when we started this. Progress has been fairly slow, which is one reason we're publishing now rather than waiting for a better stopping point.

Reverence for Life

In the original comment you wrote:

Yeah sure. I would need a full post to explain myself, but basically I think that what seems to be really important when going vegan is standing in a certain sort of loving relationship to animals, one that isn't grounded in utility but instead a strong (but basic) appreciation and valuing of the other. But let me step back for a minute.

I guess the first time I thought about this was with my university EA group. We had a couple of hardcore utilitarians, and one of them brought up an interesting idea one night. He was a vegan, but he'd been offered some mac and cheese, and in similar thinking to above (that dairy generally involves less suffering than eggs or chicken for ex) he wondered if it might actually be better to take the mac and donate the money he would have spent to an animal welfare org. And when he roughed up the math, sure enough, taking the mac and donating was somewhat significantly the better option.

But he didn't do it, nor do I think he changed how he acted in the future. Why? I think it's really hard to draw a line in the sand that isn't veganism that stays stable over time.

For those who've reverted, I've seen time and again a slow path back, one where it starts with the less bad items, cheese is quite frequent, and then naturally over time one thing after another is added to the point that most wind up in some sort of reducetarian state where they're maybe 80% back to normal (I also want to note here, I'm so glad for any change, and I cast no stones at anyone trying their best to change). And I guess maybe at some point it stops being a moral thing, or becomes some really watered down moral thing like how much people consider the environment when booking a plane ticket.

I don't know if this helps make it clear, but it's like how most people feel about harm to younger kids. When it comes to just about any serious harm to younger kids, people are generally against it, like super against it, a feeling of deep caring that to me seems to be one of the strongest sentiments shared by humans universally. People will give you some reasons for this i.e. "they are helpless and we are in a position of responsibility to help them" but really it seems to ground pretty quickly in a sentiment of "it's just bad".

To have this sort of love, this commitment to preventing s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On-Ramps for Biosecurity - A Model, published by Sofya Lebedeva on December 14, 2023 on The Effective Altruism Forum.

Thank you to the following people for reviewing: @Lin BL @Tessa @Max Görlitz @Gregory Lewis @James Smith, Sandy Hickson & @Alix Pham

TL:DR

Getting a full-time role in biosecurity is hard

Seeing a path to get there can be even harder

I propose a model to think about on-ramps into biosecurity & provide a few use cases for it depending on the background you are coming in with.

I provide an overview of how different organisations in this space fit into the model.

If you are an undergrad start here.

A common problem

When I first heard about biosecurity I was excited by the

80,000 hours podcast and impressed by the work of Kevin Esvelt, RAND and NTI. Even though I was studying molecular biology, a seemingly relevant subject I couldn't see a way for me to get involved and to find a full-time role in this field. The gap between hearing about biosecurity and working full-time in biosecurity felt huge.

Figure 1: The gap between hearing about biosecurity and working full-time in the field.

A proposed on-ramp model

Through my experiences with reading groups, UC Berkeley EA, SERI BITS and now the Oxford Biosecurity Group I have found that working on short, object-level, scalable projects fills this gap. And since I get questions of how to fill the gap from others new to the field I made a model to explain my thoughts.

Figure 2: Proposed model for On-Ramps into Biosecurity.

Using the model

Below I outline some touch points that people have with various organisations in the biosecurity space. It's important to note that this model is not always linear. It's important to question your assumptions at every stage and the "stages" themselves can be more fluid.

Hear about it (0 - 10 hours)

This stage can be passive or active depending on your timeline. Note that a lot of the 'hear about it' resources can also be 'learn about it' resources if they are used for more in-depth research at a later stage.

80,000 Hours

EA Forum (hehe)

GCBR Organization Updates Newsletter

Biosecurity newsletters you should subscribe to

University Groups

Your local EA Group

Learn about it (10 - 40 hours)

This stage usually takes around 1-2 months and is more passive.

List of Short-Term (<15 hours) Biosecurity Projects to Test Your Fit

Reading groups at your university

Reading groups at your local EA Group

Find peers (at a similar career stage to you and you can exchange ideas with)

Find mentors (who can help you deliberate between next steps in your career)

Find experts (who can help you deliberate on technical differences between projects and provide insights into specific sub-fields)

Taking to relevant people in the field, building a network

BlueDot Impact

Biosecurity Fundamentals

Project Work (40 - 100 hours)

This stage usually takes around 2-3 months and is more active. You are encouraged to continue building out your network of peers, mentors and experts and possibly to form your working group to think about these concepts. However my suggestion would be to do project work as a part of some formal group/institution if possible, to make sure that you work on something valuable.

Biosecurity Working Groups

Oxford Biosecurity Group

Wisconsin Biosecurity Initiative

Cambridge Biosecurity Group (contact:

[email protected])

Nordic Biosecurity Group (contact: Johan Täng)

Next Generation for Biosecurity Competition

BlueDot Impact

Biosecurity Fundamentals (second part of the course)

Mentorship Programs

Magnify Mentoring

IFBA Global Mentorship Program

UNODA Biosecurity Diplomacy Workshops

Short-term, full-time fellowships

Stanford Existential Risks Initiative (SERI)

Existential Risk Alliance (ERA) Cambridge Fellowship

Swiss Existential Risk Initiative (CHERI)

Full-Time Work (100 hours +)

A more extensiv...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bayesian Injustice, published by Kevin Dorst on December 14, 2023 on LessWrong.

(Co-written with Bernhard Salow)

TLDR:

Differential legibility is a pervasive, persistent, and individually-rational source of unfair treatment. Either it's a purely-structural injustice, or it's a type of "zetetic injustice" - one requiring changes to our practices of inquiry.

Finally, graduate admissions are done. Exciting. Exhausting. And suspicious.

Yet again, applicants from prestigious, well-known universities - the "Presties", as you call them - were admitted at a much higher rate than others.

But you're convinced that - at least controlling for standardized-test scores and writing samples - prestige is a sham: it's largely money and legacies that determine who gets into prestigious schools; and such schools train their students no better.

Suppose you're right.

Does that settle it? Is the best explanation for the Prestie admissions-advantage that your department has a pure prejudice toward fancy institutions?

No. There's a pervasive, problematic, but individually rational type of bias that is likely at play. Economists call it "statistical discrimination" (or "screening discrimination").

But it's about uncertainty, not statistics. We'll call it Bayesian injustice.

A simplified case

Start with a simple, abstract example. Two buckets, A and B, contain 10 coins each. The coins are weighted: each has either a or a chance of landing heads when tossed. Their weights were determined at random, independently of the bucket - so you expect the two buckets to have the same proportions of each type of coin.

You have to pick one coin to bet will land heads on a future toss.

To make your decision, you're allowed to flip each coin from Bucket A once, and each coin from Bucket B twice. Here are the outcomes:

Which coin are you going to bet on? One of the ones (in blue) that landed heads twice, of course! These are the coins that you should be most confident are weighted toward heads, since it's less likely that two heads in a row was a fluke that that one was.

Although the proportions of coins that are biased toward heads is the same in the two buckets, it's easier to identify a coin from Bucket B that has good chance to land heads. As we might say: the coins from Bucket B are more legible than those from Bucket A, since you have more information about them.

This generalizes. Suppose there are 100 coins in each bucket, you can choose 10 to bet on landing heads, and you are trying to maximize your winnings. Then you'll almost certainly bet on only coins from Bucket B (since almost certainly at least 10 of them will land HH).

End of abstract case.

The admissions case

If you squint, you can see how this reasoning will apply to graduate admissions. Let's spell it out with a simple model.

Suppose 200 people apply to your graduate program. 100 are from prestigious universities - the Presties - and 100 are from normal universities - the Normies.

What your program cares about is some measure of qualifications, qi, that each candidate i has. For simplicity, let's let qi = the objective chance of completing your graduate program.

You don't know what qi is in any given case. It ranges from 0-100%, and the committee is trying to figure out what it is for each applicant. To do so, they read the applications and form rational (Bayesian) estimates for each applicant's chance of success (qi), and then admit the 10 applicants with the highest estimates.

Suppose you know - since prestige is a sham - that the distribution of candidate qualifications is identical between Presties and Normies. For concreteness, say they're both normally distributed with mean 50%:

Each application gives you an unbiased but noisy signal, 𝞱i, about candidate i's qualifications qi.[1]

Summarizing: you know that each Prestie and Normie c...