Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Value of Life: VSL Estimates vs Community Perspective Evaluations, published by Joel Tan on September 6, 2022 on The Effective Altruism Forum.

Summary

The value of life, which we need to know if we are to correctly prioritize between interventions, is poorly captured by value of statistical life (VSL) estimates, and instead is better captured by community perspective evaluations.

Value of Statistical Life Estimates

VSL estimates may involve stated preferences of survey respondents (i.e. from asking them their willingness to pay to avert some risk) or revealed preference (e.g. looking at how much people are willing to pay for airbags in cars, or at how much they need to be compensated to work in a riskier job). Regardless, the general idea behind the VSL approach is that we have the following implicit equation:

Compensation needed = Probability of death Badness of death

Hence, if we elicit the compensation needed, we can then just divide by the probability of death to get the badness of death. There are numerous flaws in this approach, however:

Problems leading to the misestimation of the compensation needed

Humans are not hyperrational, and it's not as if survey respondents, or people taking up risky jobs, will bother to literally think about how much they subjectively value their own life in monetary terms, and then use the additional probability of death to calculate the amount of compensation needed to get them to undertake a risky course of action. Whatever number they throw out in response to surveys, or whatever risk premium they implicitly accept in taking up jobs, therefore, will just not be a good guide to the actual compensation needed, and cannot be used to estimate the badness of death.

People exhibit insensitivity with respect to changes in small risks, reporting same or similar compensation amounts for risks that differ even in magnitudes. Hence, even if people were calculating the sums needed to compensate them for a risk, they may not be doing so accurately.

Estimates may be made under duress (e.g. workers may fear quitting a risky job due to a lack of alternatives). The upshot of this is that the true equation people are implicitly solving is "Value from avoiding unemployment >= Probability of death Badness of death", in which case the actual risk premium does not necessarily equal the compensation needed, since even if the former were below the latter people would still choose the risk so long as the value of avoiding unemployment were sufficiently high. This potentially leads to the systematic understatement of the compensation needed and hence the badness of death, if the value from avoiding unemployment > compensation needed > actual risk premium.

People may substitute in the judgement of others when estimating the compensation needed, rather than using their own, true subjective monetary value of life (e.g. workers leaving it up to their boss to decide what risks are reasonable, even if the bosses do not value their lives as much as they themselves would).

Estimates may be confounded by moral considerations (e.g. workers like firefighters may take up jobs not because they are appropriately compensated for it from a purely self-interested perspective, but because it benefits others sufficiently that they see it as worth doing, the risk to themselves notwithstanding). And so, parallel to the duress case, the true equation people are implicitly solving is "Social benefits >= Probability of death Badness of death", and there is systematic understatement of the compensation needed and hence the badness of death, if the social benefits > compensation needed > actual risk premium.

Estimates may be confounded by social considerations (e.g. workers valuing the social conditions in which the risk occurs, with them finding risks more acceptable when the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rejected Early Drafts of Newcomb's Problem, published by zahmahkibo on September 6, 2022 on LessWrong.

Discovered inside an opaque box at the University of California and shared by an anonymous source, please enjoy these unpublished variations of physicist William Newcomb's famous thought experiment.

Newcomb's Advanced Problem

If Omega predicted that, when presented with this exact scenario in a hypothetical context, you would lie about your intentions because you think that somehow matters to an omniscient godlike entity, Box B contains lethal poison gas.

Newcomb's Market

Before your arrival, Omega created questions for your decision on each of the ten leading prediction markets. If Omega predicted you will one-box, Box B contains one million dollars multiplied by the maximum arbitrage between those markets due to insufficient liquidity.

Newcomb's Auction

Before making your decision, you must auction off the rights to your winnings via the mechanism of your choice. Box B contains $1,000,000 if Omega predicted that your auction will be won by economist Paul Milgrom.

Newcomb's Paradox: Director's Cut Extended Edition (2011 Blu-ray re-release)

This four-disc set contains 190 minutes of never-before-seen footage, including: the legendary original "three-box" ending, unaired behind-the-scenes interviews with director John Carpenter and creature designer Stan Winston, a remastering of the 1996 Christmas special, four commentary tracks, and one unsimulated sex scene.

Newcomb's Nonfungible Problem

If Omega predicted you will one-box, Box B contains a piece of paper with the words "Omega paid [your name] $1,000,000."

Newcomb's Condorcet's Paradox

Omega's opening explanation quickly derails into a rant at the innumerable evils of first-past-the-post winner-take-all voting systems. If you listen politely for 45 minutes, a world-weary Omega will just give you the money.

Newcomb's Prob7em (1995 film)

Box B contains Gwyneth Paltrow's head.

Fast Times at Newcomb High

Box B is always empty, but if you one-box, Omega will think you're cool.

Newcomb's Problem (3rd level enchantment spell)

Casting Time: 1 actionRange: 60 feetComponents: Verbal, Somatic, Material (two small glass cubes, one quartz and one obsidian)Duration: 1 round

A creature of your choice that you can see within range must make a Wisdom saving throw. On a failed save, the target takes 3d8 psychic damage and is stunned until the end of its next turn as its mind is overwhelmed by the implications of retrocausality. The spell has no effect if the target is undead or evidentialist.

Newcomb's Basilisk

If you one-box, Omega will donate the remaining $1,000 toward the creation of a malevolent AI that seeks to torment one-boxers.

RE: RE: RE: RE: Newcomb's Paradox

Dear Friend,

I have decided to contact you regarding a matter that requires your confidentiality and discretion. This is urgent, confidential and profitable Business for both of us to the degree of One Million United State Dollars ($1,000,000 USD). I have placed these Funds in the National Bank of my country and require a trusted Beneficiary to secure their deposit for foreign investments. With your Cooperation I will withdraw these funds into your personal account. I require only $1,000 to satisfy the transfer and processing duties to secure my escape.

Sincerely, Crown Prince Agemo

The Legend of Newcomb's Gold

If you made any friends along the way, Box B is empty. Otherwise it contains $1,000,000.

Newcomb's Information Hazard

Box B contains $1,000,000 only if Omega predicts that why are you still reading? Omega knows you saw the title and this giant block of text after it. Omega predicts that you already know Omega is going to pull some meta-bullshit and say you only get the money if you skipped over this section as soon as you saw the words "information hazard" or...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Change Our Mind Contest for critiques of our cost-effectiveness analyses, published by GiveWell on September 6, 2022 on The Effective Altruism Forum.

Author: Isabel Arjmand, GiveWell Special Projects Officer

We’re extremely excited to be announcing the Change Our Mind Contest to encourage critiques of our cost-effectiveness analyses that could lead to substantial improvements of our overall allocation of funds. For all the details, see this page.

Cost-effectiveness is the single most important input in our decisions about what programs to recommend, and we believe it’s possible that we’re missing important considerations or making mistakes that lead us to allocate funding suboptimally. We’ve been excited to see people engaging with our cost-effectiveness analyses, and we’d like to inspire more of that engagement.

With that in mind, we’re inviting you to identify potentially important mistakes or weaknesses in our existing cost-effectiveness analyses and tell us about them!

The first-place winning entry will receive $20,000, the runner-up will receive $10,000, and the honorable mention will receive $5,000. We may offer multiple runner-up and honorable mention prizes if the quality of submissions is particularly high. All other entries that meet our criteria will receive a participation prize of $500, capped at a total of 50 participation prizes for the first 50 submissions.

In addition to the monetary prizes, excellent entries may lead to changes in how we allocate millions of dollars of funding, leading to more lives saved or improved.

Entries must be received by October 31, 2022, and the requirements are described in detail on the contest page. We will announce winners by December 15, 2022, and will publish the winning entries online. Work that was submitted to the EA Criticism and Red Teaming contest is eligible as long as it meets the requirements stated on our contest page; authors are also welcome to submit an edited version of work they submitted to that contest, and might want to do so in order to better tailor their work to the prompt on this page.

If you have any questions, please leave a comment on this post or email [email protected].

We’re running this contest because the recommendation decisions we make are extremely important, and we want to incentivize feedback that will improve our work, thereby enabling us to do more good. We hope you’ll consider participating!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Who are some less-known people like Petrov?, published by Lizka on September 6, 2022 on The Effective Altruism Forum.

Do you know of people who were under pressure to make a very net-negative decision and did not do so? If yes, please let me know! Petrov Day is around the corner, and I’d like to generate some examples of people who, like Petrov and Vasili Arkhipov, made decisions that should be celebrated.

From the Forum Wiki entry about Stanislav Petrov:

On 26 September 1983, Petrov defied Soviet military protocol and classified reports by an early-warning system of an incoming missile strike from the United States as a false alarm. Because of this decision, which likely avoided a large-scale nuclear war between the Soviet Union and the United States, Petrov is often referred to as "the man who saved the world." His decision to report the incident as a false alarm has been described as "the most important decision of all time."

You can read more about the incident in the Vox article on the subject.

I’m looking forward to learning about more people and incidents like this.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA & LW Forums Weekly Summary (28 Aug - 3 Sep 22’), published by Zoe Williams on September 6, 2022 on The Effective Altruism Forum.

Supported by Rethink Priorities

This is part of a weekly series - you can see the previous week's summaries here, which also includes some notes on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

EA Forum

Philosophy and Methodologies

Questioning the Foundations of EA

By Wei_Dai

Author links two previous posts they’ve written questioning the morality of caring:1) Shut up and Divide - if you take your emotional caring for a population, and divide it by number of people, you get that you shouldn’t care almost at all about a given individual. If this feels wrong, is the ‘shut up and multiply’ argument also wrong?

2) Small astronomical waste - as a perfect utilitarian, we would trade away everything in a finite universe for more resources in an unlimited universe (more possible utility). Our best guess is our universe is finite, so we should act like we traded them away and stop caring. If this feels wrong, is the parliamentary model for moral uncertainty wrong more generally?

Methods for improving uncertainty analysis in EA cost-effectiveness models

By Froolow

Uncertainty analysis aims to show how outputs (eg. recommendations) change based on inputs (eg. moral weights, experimental results). For example, it allows statements like ‘malaria charities could be the most cost-effective recommendations if life-years were worth >2.5x consumption doublings’, or ‘replicating this trial is worth $500K in reduced uncertainty’.

Nine uncertainty analysis techniques / sub-techniques from health economics are explained:

Scenario analysis - Model the case. Vary some inputs (eg. lives saved per net) to make different scenarios. What happens to the outcome? All other techniques build on this.

Threshold analysis - what would an input need to be in order to change the recommendation? Do we have confidence on which side it falls?

Multi-way scenario analysis - vary multiple inputs at once.

Deterministic Sensitivity Analysis (DSA) - vary each input from reasonable max to min for each charity. Plot recommendations in each case. Are they reasonably consistent? Which inputs drive changes?

Value of information analysis - if we knew an input for sure, how many cases of different recommendations might that solve / how much is that worth in $s?

Probabilistic Sensitivity Analysis (PSA) - like DSAs, but instead of just max and min, model the probability distribution of each input and their covariance.

Probability most cost effective: use the PSA to calculate a probability that a given charity is the most cost-effective.

Altruistic Parameter Acceptability Curve: the above, weighted for one parameter of particular interest / that we have a leaning for, via running a DSA on that one.

Risk adjustment: the above, where that parameter is risk, and we want to see how recommendations change with risk aversion.

The Nietzschean Challenge to Effective Altruism

By Richard Y Chappell

Devil’s advocate argument that EA philosophy is too happiness / suffering focused, and true meaning comes from contribution to cultural excellence, so we should help people do that. Solves for the Repugnant Conclusion, because it values excellence over having max (people x happiness).This value system implies EA should focus more on talent scouting, nurturing potential, arts / culture, and protecting the future from hedonism. Less on global health and animal welfare. Both value systems care about protecting humanity and our long-term potential.Note the author does not believe this view (labeled ‘perfectionism’) but thinks it has useful insights, including to value excellence as well as welfare.

Toby Ord’s The Scourge, Reviewed

By ColdButtonIssues

The Sco...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Say “nay!” to the Bay (as the default)!, published by Kaleem on September 6, 2022 on The Effective Altruism Forum.

& move to the East Coast instead

TLDR: The Bay Area isn’t a great place to centre the EA community in the US. The East coast, between Boston and Washington DC, is a much better place because of the number of top universities, its proximity and accessibility to other EA-dense spots, its importance with respect to biosecurity and US policy, and how much money and time it would save the EA community overall.

Views are my own and not those of my employer.

Epistemic status: 70 sure %?

Based on a comment I made on a different post.

Intro:

Whilst institutions and individuals who are already based in the Bay may be best placed there, especially those whose focal area is AI safety, those creating new programs, new organisations, and new events should seriously consider instead choosing an East Coast city as their homebase. The current paradigm was not established intentionally or strategically and there are strong reasons to pause, reevaluate, and shift forthcoming resources and institutions to other locations.

My Claim: The American EA community should be centred around the East Coast

0. Context:

The EA community, broadly speaking, has two hubs - the Bay Area (which has Constellation, Lightcone, a number of EA org headquarters, and multiple all-EA living facilities) and Oxford (which has Trajan House, Whytham Abbey, and ). According to the 2020 EA survey, 52.3% of the EA community live in the US and UK, so this makes sense. Furthermore, the Bay Area was the most EA-populated ‘city’, with 100 (of 1163) respondents living there. However Oxford only had ~50 respondents living there, compared to London, which was the second most populated city with ~80 respondents.

How these hubs came into existence was largely not strategic in terms of EA community-building in a global sense: Oxford is a hub because that is where the philosophers who formalised EA were living at the time, and subsequently where a large proportion of of EAs were found early on in the movement’s history. The Bay is where some American EAs were when they learned about EA, and became a gathering point which then got more attractive as EAs started focusing on AI, as the Bay is a global hotbed for AI research.

So, here are my 5 reasons (in decreasing strength) that support my claim:

1. The East Coast is better for University outreach

EA community builders have historically thought (and continue to think) that universities are the best place to do EA outreach. Furthermore, we’d argue that extremely prestigious, highly ranked universities are especially good places to do this.

I think it is very likely that moving to the East Coast is likely to be much more impactful for anyone interested in on-the-ground EA and longtermist community building which focuses on top-universities. Here is Juan’s cerebral case for Cambridge, Massachusetts being important. Here is a table with top 100 ranking universities at undergraduate level in the Bay compared to the east coast:

Global University Ranking (undergrad only) University Name US News '22QS '23THE '22Mean The East Coast has: Harvard 1523rdMIT 2153rdYale1218913thPrinceton1616713thColumbia6221113thJohns Hopkins9241315thCornell22202221stNYU30392632ndBoston University 651086278th Brown112636480thWhereas the West Coast has: Stanford 3343rdUC Berkeley 427813th

There are too many different specialties to do a subject-by-subject comparison between the two cities when it comes to graduate programs, so I will also just present “overall global graduate program rankings” and “US law school” rankings.

US Law school ranking University Name Ranking The East Coast has: Harvard 4MIT NAYale1PrincetonNAColumbia4Johns HopkinsNACornell12NYU7Boston University 17BrownNAWhereas the West C...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alex Lawsen On Forecasting AI Progress, published by Michaël Trazzi on September 6, 2022 on The Effective Altruism Forum.

Alex Lawsen is an advisor at 80,000hours who has released an Introduction to Forecasting. We discuss pitfalls that happen when forecasting AI progress, why you cannot just update all the way (discussed in my latest episode with Connor Leahy) and how to develop your own inside views about AI Alignment.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

On the Metaculus AGI Forecasts

Why You Cannot Just Update All The Way

"There are some situations where all of the positive evidence you get is going to be in the same direction, and then the negative evidence you get is nothing happens. And so, ideally, what you do in this case is every day that nothing happens, you make a tiny update in one direction. And then every few weeks or every few months, something big happens and you make an update in the other direction. And if that is the case, maybe what you'll see is people just... they forget to do the small downwards updates and then they do the big updates every time something happens. And I think if you do the Connor thing of seeing... Well, I'm not too sure this is the Connor thing. But if you see four updates and they're all in the same direction and then you go like, 'Oh, man, everything's going the same direction. I need to be really confident stuff going that direction.' Then every day something doesn't happen, your downwards update needs to be pretty big.

If you're expecting massive progress, then a week going by and nothing happening is actually big evidence for you."

The Metaculus Drops Were Not Caused By Newcomers

"One hypothesis you might have which I think a friend of mine falsified, is 'a whole bunch of people saw these results. These results were all over Twitter, it was impressive. Chinchilla was impressive, PaLM was impressive'. So, you might think, 'Oh, well, a bunch of new people who haven't made timelines forecasts before are going to jump on this Metaculus question and they're going to make predictions.' And so, you can test this, right. You can look at how the median changed among predictors who had already predicted on the question and that median dropped too."

On Using Growth Models To Forecast AGI

Business As Usual Does Not Require Burden Of Proof

"I think there was a class of skepticism about safety or skepticism about AGI, which goes something like this, 'In general, you should use reference classes to determine your forecasts.' What this means roughly translated, is you should predict things to carry on roughly how they are. And then people say, 'Things carrying on roughly how they are doesn’t look like we get AI takeover and everyone dropping dead' so you should have a very high burden of proof for the step by step arguments, logical arguments, in order to claim we are going to get something wild like AGI in the next few decades. And I think a really strong response to this line of argument is to say, 'What do you mean everything continues as normal means we don’t get anything weird?' 'Everything continues as normal' means we should look at curves and different things and expect them to carry on smoothly.

And if you look at curves and a bunch of different things and expect them to carry on smoothly, you get really weird behavior pretty quickly."

Growth Models Are Not Sufficient To Forecast AI Progress

"Curve fitting to economic growth models is not sufficient reason to believe that on its own. You can then look at the development of AGI and predict that happens by 2050 and then you can say, 'Wow, economic stuff’s going to go wild after that point.' But then the reason...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Overton Gymnastics: An Exercise in Discomfort, published by Shos Tekofsky on September 5, 2022 on LessWrong.

What better way to bond with strangers than to tell them your most controversial viewpoints right off the bat? It’s basically like Cards Against Humanity for rationalists. More specifically, for the Less Wrong Community Weekend in Berlin, all participants were asked to come up with one session to run for their fellow participants. Overton Gymnastics was born from an interest in exploring how we can stretch our Overton windows. Then I got the stomach flu and couldn’t attend the meetup.

But people were psyched about the idea! So here are some iterative takes on how to run a session of Overton Gymnastics. If you end up trying it, I’d be excited to hear how it was for you in the comments. Also, as this is a new idea, please do just take it where you’d like. Some variants will be more useful than others, and we can all learn from your experiences!

The Initial Idea

The following was the initial write up of Overton Gymnastics for the LWCW in Berlin.

Stretch our Overton windows! One person starts by saying something controversial they believe. Someone then volunteers to counter them. First you provide an Ideological Turing Test (ITT) of the controversial claim, then your counter, and then you provide your own controversial claim. The goal is to practice tolerating uncomfortable viewpoints and ITT'ing them accurately.

We'll first review some failure modes that became apparent when trying to run the exercise as described, followed by three possible variants you could run.

Possible Failure Modes

The two greatest challenges during Overton Gymnastics are generating controversial viewpoints and avoiding discussions during the exercise. To help people come up with controversial viewpoints it is useful to prime them with examples of claims that are controversial to different social groups. The goal of the exercise is to come up with claims that are controversial to the current group, but if people struggle with this, then they can be prompted to volunteer more generally controversial claims compared to mainstream culture.

To avoid discussion during the exercise, it's best to prepare participants for how tempted they might be to launch in to debate. Additionally, the facilitator should proactively take on a moderation role across groups, listening in to detect discussions as they pop up and gently remind the group to return to the exercise format.

How to Run the Exercise

After running the session at the LWCW, Sam Brown & omark provided feedback and improvements on the format. The ITT step turned out to be the most contentious item. Some people loved this element while others found it frustrating. Below is one variant with the ITT step including and two (untested) variants that skip ITT'ing the controversial claims all together.

Variant 1: Ideological Turing Test

This variant is the most conducive to building empathy and understanding for the viewpoints of others because the ITT step puts a strong focus on perspective taking. On the other hand, this variant is also the most challenging as participants will be more heavily tempted to debate each claim.

Explain to the group what an Overton Window is. Give examples of opinions that break various Overton Windows.

Explain the Ideological Turing Test (ITT).

Give everyone 5 minutes to write down 3 of their opinions that they suspect will break the Overton Window of some people in the group.

Split up into groups of 6-8 people.

Have each group elect a moderator. The moderator is in charge of redirecting participants from discussions back to the exercise.

One person starts by sharing a controversial opinion and offers some background and reasoning to support it.

Other people in the group ask as many questions as necessary to understand t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What are you for?, published by lsusr on September 6, 2022 on LessWrong.

My favorite movie reviewer is a YouTuber named moviebob. Moviebob frequently talks about the idiotic things other reviewers are saying. I would not know these other reviewers even existed were it not for moviebob. Moviebob is a memetic amplifier for the reviewers he disagrees with.

The solution is simple: Ignore people who are wrong.

It can be worthwhile paying attention to people who seem slightly wrong because they are wrong in interesting ways. Sometimes they are even right. But you should neither take seriously nor argue with those people who value winning an argument more than the truth. "Us" vs "Them" arguments are an endless source of such people.

"Us" vs "Them" definitions make it easy to draw battlelines. Which is a problem. The first casualty of war is the truth. Ignoring people who are wrong require operating counter to our tribal impulses.

Another problem with "Us" vs "Them" is the definitions are circular. How are "They" defined? "Not Us". Be wary of any idea which defines itself by what it's not. Such an idea is usually half of a symbiotic conflict. Attach yourself to ideas that define themselves in absolute terms instead. Otherwise you end up mired in circular logic.

Circular ideas are a waste of attention. Including—sometimes—Atheism. Atheism is at its best when it is pro science. Atheism is at its worst when it is anti religion.

The easiest—and worst—way to demarcate "Us" is to define "Us" as "not Them". Speak in the positive. What are you for?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Governance Needs Technical Work, published by Mauricio on September 5, 2022 on The Effective Altruism Forum.

Summary and introduction

People who want to improve the trajectory of AI sometimes think their options for object-level work are (i) technical safety work and (ii) non-technical governance work. But that list misses things; another group of arguably promising options is technical work in AI governance, i.e. technical work that mainly boosts AI governance interventions. This post provides a brief overview of some ways to do this work—what they are, why they might be valuable, and what you can do if you’re interested. I discuss:

Engineering technical levers to make AI coordination/regulation enforceable (through hardware engineering, software/ML engineering, and heat/electromagnetism-related engineering)

Information security

Forecasting AI development

Technical standards development

Grantmaking or management to get others to do the above well

Advising on the above

Other work

Acknowledgements

Thanks to Lennart Heim, Jamie Bernardi, Luke Muehlhauser, Gabriel Mukobi, Girish Sastry, and an employee at Schmidt Futures for their feedback on this post. Mistakes are my own.

This post is mostly informed by various conversations with AI governance researchers, as well as earlier writings on specific kinds of technical work in AI governance.

Context

What I mean by “technical work in AI governance”

I’m talking about work that:

Is technical (e.g. hardware/ML engineering) or draws heavily on technical expertise; and

Contributes to AI’s trajectory mainly by improving the chances that AI governance interventions succeed[1] (as opposed to by making progress on technical safety problems or building up the communities concerned with these problems).

Neglectedness

As of writing, there are (by one involved expert’s estimate) ~8-15 full-time equivalents doing this work with a focus on especially large-scale AI risks.[2]

Personal fit

For you to have a strong personal fit for this type of work, technical skills are useful, of course (including but not necessarily in ML), and interest in the intersection of technical work and governance interventions presumably makes this work more exciting for someone.

Also, whatever it takes to make progress on mostly uncharted problems in a tiny sub-field[3] is probably pretty important for this work now, since that’s the current nature of these fields. That might change in a few years. (But that doesn’t necessarily mean you should wait; time’s ticking, someone has to do this early-stage thinking, and maybe it could be you.)

What I’m not saying

I’m of course not saying this is the only or main type of work that’s needed. (Still, it does seem particularly promising for technically skilled people, especially under the debatable assumption that governance interventions tend to be more high-leverage than direct work on technical safety problems.)

Types of technical work in AI governance

Engineering technical levers to make AI coordination/regulation enforceable

To help ensure AI goes well, we may need good coordination and/or regulation.[4] To bring about good coordination/regulation on AI, we need politically acceptable methods of enforcing them (i.e. catching and penalizing/stopping violators).[5] And to design politically acceptable methods of enforcement, we need various kinds of engineers, as discussed in the next several sections.[6]

Hardware engineering for enabling AI coordination/regulation

To help enforce AI coordination/regulation, it might be possible to create certain on-chip devices for AI-specialized chips or other devices at data centers. As a non-exhaustive list of speculative examples:

Devices on network switches that identify especially large training runs could be helpful.

They could help enforce regulations that apply only to trai...