Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Kidney stone pain as a potential cause area, published by Dan Elton on May 3, 2022 on The Effective Altruism Forum.

This is a cross-post from my Substack (original post here). Note: At the Effective Altruism Global: San Francisco conference in 2017, Prof. Will MacAskill implored the audience to “keep EA weird”. As the EA movement grows, it’s important to keep EA’s original spirit of exploration alive. To help do that, I’m planning to write several articles on potential new — and weird — cause areas.

Effective altruists want to figure out how to do the most good per dollar spent. “Good” is often cached out in terms of deaths prevented or quality-adjusted life years saved (QALYs). QALYs attempt to adjust life-years for different states of health. However, the very method of QALY calculation, which typically involves surveys asking about trade-offs, might have some blind spots. For instance, if a disease state is rare, than most likely the requisite survey data for it has never been collected. The surveys themselves may have blind spots too. Consider these points from Andrés Gómez Emilsson (emphasis mine):

“Someone described the experience of having a kidney stone as ‘indistinguishable from being stabbed with a white-hot-glowing knife that's twisted into your insides non-stop for hours’. It’s likely that the reason why we do not hear about this is because (1) trauma often leads to suppressed memories, (2) people don’t like sharing their most vulnerable moments, and (3) memory is state-dependent (you cannot easily recall the pain of kidney stones .. you’ve lost a tether/handle/trigger for it, as it is an alien state-space on a wholly different scale of intensity than everyday life).”

Andres Gomez Emilsson

As Daniel Kahneman describes in his book Thinking Fast and Slow, the remembering self is different than the experiencing self. People have trouble describing and conceptualizing extreme events, either positive or negative. People also don’t like thinking about extreme negative events generally, whether they experienced them or others did. I personally sometimes notice my brain flinching away when thinking about kidney stone pain, even though I haven’t experienced it myself.

In the first part of this post I’ll go over the evidence for extreme pain events. Then, I’ll focus on kidney stones. The main reason for focusing on kidney stone pain is that over the past two years I’ve worked off-an-on on automated deep learning based software for detecting and measuring kidney stones in CT scans (see my paper in Medical Physics). So I have some expertise on the subject. Currently I am working with a radiologist at Massachusetts General Hopsital who is an expert on stone disease, Prof. Avinash Kambadakone.

Background - suffering focused ethics

“In my opinion human suffering makes a direct moral appeal, namely, the appeal for help, while there is no similar call to increase the happiness of a man who is doing well anyway.”

“Instead of the greatest happiness for the greatest number, one should demand, more modestly, the least amount of avoidable suffering for all.”— Karl Popper, The Open Society and Its Enemies (1945)

The idea that we should focus on eliminating suffering over increasing pleasure is intuitive to many people. See this recent Twitter poll from Robin Hanson:

So, I don’t think I need to spend much time here convincing people that reducing suffering should take precedent over increasing happiness. Note what I have in mind here is what is called “weakly-negative utilitarianism” which is quite different than pure negative utilitarianism, which focuses only on eliminating suffering. Readers interested in diving further into these topics should check out Lukas Gloor’s essay “The Case for Suffering-Focused Ethics”.

Background - long-tailed distributions of pleasure and pain

“...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is EA "just longtermism" now?, published by frances lorenz on May 3, 2022 on The Effective Altruism Forum.

Acknowledgements

A ginormous thank you to Bruce Tsai who kindly combed through multiple drafts and shared tons of valuable feedback. Also a big thank you to Shakeel Hashim, Ines, and Nathan Young for their insightful notes and additions.

Preface

In this post, I address the question: is EA just longtermism? I then ask, among other questions, what factors contribute to this perception? What are the implications?

1. Introduction

Recently, I’ve heard a few criticisms of EA that hinge on the following: “EA only cares about longtermism.” I’d like to explore this perspective a bit more and the questions that naturally follow, namely: How true is it? Where does it come from? Is it bad? Should it be true?

2. Is EA just longtermism?

In 2021, around 60% of funds deployed by the Effective Altruism movement came from Open Philanthropy (1). Thus, we can use their grant data to try and explore EA funding priorities. The following graph (from Effective Altruism Data) shows Open Philanthropy’s total spending, by cause area, since 2012:

Overall, Global Health & Development accounts for the majority of funds deployed. How has that changed in recent years, as AI Safety concerns grow? We can look at this uglier graph (bear with me) showing Open Philanthropy grants deployed from January, 2021 to present (data from the Open Philanthropy Grants Database):

We see that Global Health & Development is still the leading fund-recipient; however, Risks from Advanced AI is now a closer second. We can also note that the third and fourth most funded areas, Criminal Justice Reform and Farm Animal Welfare, are not primarily driven by a goal to influence the long-term future

With this data, I feel pretty confident that EA is not just longtermism. However, it is also true (and well-known) that funding for longtermist issues, particularly AI Safety, has increased. This raises a few more questions:

2.1 Funding has indeed increased, but what exactly is contributing to the view that EA essentially is longtermism/AI Safety?

(Note: this list is just an exploration and not meant to claim whether the below things are good or bad)

William Macaskill’s upcoming book, What We Owe the Future, has generated considerable promotion and discussion. Following Toby Ord’s The Precipice, published in March, 2020, I imagine this has contributed to the outside perception that EA is becoming synonymous with longtermism.

The longtermist approach to philanthropy is different from mainstream, traditional philanthropy. When trying to describe a concept like Effective Altruism, sometimes the thing that most differentiates it is what stands out, consequently becoming its defining feature.

Of the longtermist causes, AI Safety receives the most funding, and furthermore, has a unique ‘weirdness’ factor that generates interest and discussion. For example, some of the popular thought experiments used to explain Alignment concerns can feel unrealistic, or something out of a sci-fi movie. I think this can serve to both: 1. draw in onlookers whose intuition is to scoff, 2. give AI-related discussions the advantage of being particularly interesting/compelling, leading to more attention.

AI Alignment is an ill-defined problem with no clear solution and tons of uncertainties: What counts as AGI? What does it mean for an AI system to be fair or aligned? What are the best approaches to Alignment research? With so many fundamental questions unanswered, it’s easy to generate ample AI Safety discussion in highly visible places (e.g. forums, social media, etc.) to the point that it can appear to dominate EA discourse.

AI Alignment is a growing concern within the EA movement, so it's been highlighted recently by EA-aligned orgs (for example, AI S...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Information security considerations for AI and the long term future, published by Jeffrey Ladish on May 2, 2022 on The Effective Altruism Forum.

Summary

This post is authored by Jeffrey Ladish, who works on the security team at Anthropic, and Lennart Heim, who works on AI Governance with GovAI (more about us at the end). The views in the post are our own and do not speak for Anthropic or GovAI. This post follows up on Claire Zabel and Luke Muehlhauser’s 2019 post, Information security careers for GCR reduction.

We’d like to provide a brief overview on:

How information security might impact the long term future

Why we’d like the community to prioritize information security

In a following post, we will explore:

How you could orient your career toward working on security

Tl;dr:

New technologies under development, most notably artificial general intelligence (AGI), could pose an existential threat to humanity. We expect significant competitive pressure around the development of AGI, including a significant amount of interest from state actors. As such, there is a large risk that advanced threat actors will hack organizations — that either develop AGI, provide critical supplies to AGI companies, or possess strategically relevant information— to gain a competitive edge in AGI development. Limiting the ability of advanced threat actors to compromise organizations working on AGI development and their suppliers could reduce existential risk by decreasing competitive pressures for AGI orgs and making it harder for incautious or uncooperative actors to develop AGI systems.

What is the relevance of information security to the long term future?

The bulk of existential risk likely stems from technologies humans can develop. Among candidate technologies, we think that AGI, and to a lesser extent biotechnology, are most likely to cause human extinction. Among technologies that pose an existential threat, AGI is unique in that it has the potential to permanently shift the risk landscape and enable a stable future without significant risks of extinction or other permanent disasters. While experts in the field have significant disagreements about how to navigate the path to powerful aligned AGI responsibly, they tend to agree that actors that seek to develop AGI should be extremely cautious in the development, testing, and deployment process, given the failures could result in catastrophic risks, including human extinction.

NIST defines information security as “The protection of information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction.”

We believe that safe paths to aligned AGI will require extraordinary information security effort for the following reasons:

Insufficiently responsible or malicious actors are likely to target organizations developing AGI, software and hardware suppliers, and supporting organizations to gain a competitive advantage.

Thus, protecting those systems will reduce the risk that powerful AGI systems are developed by incautious actors hacking other groups.

Okay, but why is an extraordinary effort required?

Plausible paths to AGI, especially if they look like existing AI systems, expose a huge amount of attack surface because they’re built using complex computing systems with very expansive software and hardware supply chains.

Securing systems as complex as AGI systems is extremely difficult, and most attempts to do this have failed in the past, even when the stakes have been large, for example, the Manhattan Project.

The difficulty of defending a system depends on the threat model, namely the resources an attacker brings to bear to target a system. Organizations developing AGI are likely to be targeted by advanced state actors who are amongst the most capable hackers.

Even though this is a challengin...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Information security considerations for AI and the long term future, published by Jeffrey Ladish on May 2, 2022 on LessWrong.

Summary

This post is authored by Jeffrey Ladish, who works on the security team at Anthropic, and Lennart Heim, who works on AI Governance with GovAI (more about us at the end). The views in the post are our own and do not speak for Anthropic or GovAI. This post follows up on Claire Zabel and Luke Muehlhauser’s 2019 post, Information security careers for GCR reduction.

We’d like to provide a brief overview on:

How information security might impact the long term future

Why we’d like the community to prioritize information security

In a following post, we will explore:

How you could orient your career toward working on security

Tl;dr:

New technologies under development, most notably artificial general intelligence (AGI), could pose an existential threat to humanity. We expect significant competitive pressure around the development of AGI, including a significant amount of interest from state actors. As such, there is a large risk that advanced threat actors will hack organizations — that either develop AGI, provide critical supplies to AGI companies, or possess strategically relevant information— to gain a competitive edge in AGI development. Limiting the ability of advanced threat actors to compromise organizations working on AGI development and their suppliers could reduce existential risk by decreasing competitive pressures for AGI orgs and making it harder for incautious or uncooperative actors to develop AGI systems.

What is the relevance of information security to the long term future?

The bulk of existential risk likely stems from technologies humans can develop. Among candidate technologies, we think that AGI, and to a lesser extent biotechnology, are most likely to cause human extinction. Among technologies that pose an existential threat, AGI is unique in that it has the potential to permanently shift the risk landscape and enable a stable future without significant risks of extinction or other permanent disasters. While experts in the field have significant disagreements about how to navigate the path to powerful aligned AGI responsibly, they tend to agree that actors that seek to develop AGI should be extremely cautious in the development, testing, and deployment process, given the failures could result in catastrophic risks, including human extinction.

NIST defines information security as “The protection of information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction.”

We believe that safe paths to aligned AGI will require extraordinary information security effort for the following reasons:

Insufficiently responsible or malicious actors are likely to target organizations developing AGI, software and hardware suppliers, and supporting organizations to gain a competitive advantage.

Thus, protecting those systems will reduce the risk that powerful AGI systems are developed by incautious actors hacking other groups.

Okay, but why is an extraordinary effort required?

Plausible paths to AGI, especially if they look like existing AI systems, expose a huge amount of attack surface because they’re built using complex computing systems with very expansive software and hardware supply chains.

Securing systems as complex as AGI systems is extremely difficult, and most attempts to do this have failed in the past, even when the stakes have been large, for example, the Manhattan Project.

The difficulty of defending a system depends on the threat model, namely the resources an attacker brings to bear to target a system. Organizations developing AGI are likely to be targeted by advanced state actors who are amongst the most capable hackers.

Even though this is a challenging problem requiring...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: P(utopia) is more important than P(doom) and this could have important strategic implications, published by Joshua Clymer on May 2, 2022 on The Effective Altruism Forum.

Edit: I'm assuming total utilitarian values and the basic ideas behind longtermism without providing any justification for them. If you do not have these values or disagree with longtermism, these arguments probably won't apply to you.

I am concerned that the longtermist community is too down-side focused and this could result in people not thinking enough about increasing the probability of utilitarian utopia.

Claim 1: the probability that superintelligent AGI has utilitarian values dominates EV calculations.

There are roughly three ways I can see superintelligence going down:

The superintelligence is misaligned and we all die.

The superintelligence is aligned to non-utilitarian values (probably normal human values). Perhaps humans end up colonizing other solar systems. Or maybe they all decide to enter some kind of simulation – but they don’t use every joule and kilogram available to produce utility.

The superintelligence is aligned to utilitarian values – i.e. every Joule in every reachable galaxy is used to produce happy, meaningful lives. Imagine Dyson spheres surrounding every star to support countless brains in vats or simulated minds.

Evaluating outcome 1:

I'll ignore the moral relevance of the superintelligence itself since it is present in all three outcomes. If the superintelligence is unaligned, I don’t see any reason to expect it to do anything particularly utilitarian (or anti-utilitarian), so I will value this outcome at 0.

Evaluating outcome 2:

I'll be generous and say that humans will populate all reachable galaxies and there will be 10 billion happy humans per star.

Evaluating outcome 3:

I'll be pessimistic and assume that the hard problem of consciousness can’t be solved and the superintelligence must construct biological brains to ensure that real experiences are being produced. A brain consumes 20 joules per second and a star produces something like 10^26 joules per second, so a single star can support 10^25 brains. 10^25 brains would require a lot of matter, but energy can be converted into matter. There are 10^16 Joules in a kg and a brain weighs ~ 1 kg which means it would take 10^9 seconds = 32 years for a star to produce enough matter to support its full brain capacity, which is no time at all in the scheme of things. So, outcome three produces something like 10^25 happy humans per star (or many more).

So, the expected utility of the universe should look something like:

Utopia is where almost all of the value is.

Claim 2: It is not obvious that reducing the probability of extinction is the best way to increase the probability of utopia.

According to the previous section, almost all of the value in reducing x-risk comes from indirectly increasing the probability of utopia. Imagine a scale that represents the probabilities of the three outcomes with two handles: the doom handle, and the utopia handle:

When you pull either handle, the rest of the chart scales proportionally.

Some interventions that mostly just pull the doom handle:

The majority of AI alignment research agendas: interpretability, inner-alignment, learning from human feedback, etc.

Increasing awareness about the alignment problem.

Some interventions that pull the utopia handle:

Directing EA community-building efforts at people who could have influence over AGI.

Most longtermist governance research.

Investing in capabilities at companies where there is more EA influence (e.g. anthropic)

Trying to get EAs into positions of influence in AI companies or the government.

Comparing scale

Note that the value of pulling the doom handle depends a lot on the ratio between the blue and yellow areas.

x = p(normal values)/p(ut...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Job: EA Office Manager, published by Jonathan Michel on May 2, 2022 on The Effective Altruism Forum.

Summary

I think being an Office Manager at an EA office can be a really impactful job, and I would like to share my experience and give some advice. Part of the reason I’m writing this is that I’m the new Head of Property at the CEA Operations Team, but my previous job was being the office manager, and I’m hiring someone to replace me. If you’re interested, please apply here or get in touch with me if you have questions.

How much you should trust me? I’m biased, but I’m trying to be honest and just share my experience.

Why am I writing this?

There has already been a bit of past discussion of ops on the forum. The reason I want to add to this pile of writing about operations is that I want to advocate/give context on the specific role of an EA Office Manager — one that oftentimes people underestimate in its importance. There has been some talk regarding EA Hubs and where we should start new ones. I think that EA Hubs can be very impactful and that a great office manager is a crucial component. An outstanding EA office which makes people more productive and happy needs an outstanding office manager.

My Background

Overall: I had a lot of experience with EA (I ran a local group for over four years, was well-read, and organised a bunch of events), and did a lot of volunteering through which I built operations skills.

I did a lot of volunteer work for the German EA community such as organizing fellowships, talks and a retreat.

I co-founded two NGOs (one focused on COVID relief and the other on cellular agriculture)

I did a bunch of volunteering for GFI, ProVeg, and an internship for www.effektiv-spenden.org

Day in the life

Last year, I started as the Office Manager of the Oxford EA office, Trajan House. Trajan House currently accommodates the Centre for Effective Altruism, the Future of Humanity Institute, the Global Priorities Institute, the Forethought Foundation, the Centre for the Governance of AI, the Global Challenges Project, Our World in Data, and a number of people working at other EA organisations (such as Rethink Priorities, HLI, LEEP, and OpenPhil).

At the moment, around 80 EA professionals work at Trajan House, and this number is growing. If you want to get a better sense of Trajan House (including some photos) you can see the office guide here.

Until very recently, the office team consisted of me (as the office manager), and employees of Oxford University working in the reception area and in facilities management. One month ago two office assistants joined my team, so we now have two additional FTEs helping to run the office. As described above, I’m now transitioning out of the office manager role, but the below outlines my week in the position.

How I spent my time:

40% - Expanding, changing and optimising the office set-up (including thinking about how we can further expand and improve the services we provide)

20% - Developing the culture and community aspects of the office (e.g. by planning events)

20% - Processing direct requests like “Can I get a MacBook charger, please?”, “Do we have spare copies of The Precipice?”

10% - managing the Office Assistants and liaising with the Facilities Management team

5% - Processing requests of individuals or organisations for office space

For a more tangible sense of what I did, here are some specific things I did in the last couple of months:

Changing the acoustics (the “soundscape”) of our cafeteria (getting different quotes done, thinking about the interior design of the space, liaising with the contractors implementing it)

Finding a new caterer (researching different companies, work-trialling them, negotiating a contract in cooperation with our lawyers, having regular check-ins to ensure quality and improve their s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New study on whether animal welfare reforms affect wider attitudes towards animals, published by Jamie Harris on May 2, 2022 on The Effective Altruism Forum.

A new study called “The Effects of Exposure to Information About Animal Welfare Reforms on Animal Farming Opposition: A Randomized Experiment” has been published in Anthrozoös.

As context, some advocates worry that learning about farmed animal welfare reforms can increase complacency with factory farming. (More discussion here.) We wanted to put that to the test.

The experiment found that, unlike information about current farming practices, which somewhat increased animal farming opposition, providing information about animal welfare reforms had no significant effect on animal farming opposition, but we found evidence of indirect effects via some of the mediators we tested.

Written by me, Ali Ladak (Sentience Institute), and Maya Mathur (Assistant Professor, Stanford University).

ABSTRACT

There is limited research on the effects of animal welfare reforms, such as transitions from caged to cage-free eggs, on attitudes toward animal farming. This preregistered, randomized experiment (n = 1,520) found that participants provided with information about current animal farming practices had somewhat higher animal farming opposition (AFO) than participants provided with information about an unrelated topic (d = 0.17). However, participants provided with information about animal welfare reforms did not report significantly different AFO from either the current-farming (d = −0.07) or control groups (d = 0.10). Although these latter effects on AFO were small and nonsignificant, they appeared to be mediated by changes in perceived social attitudes toward farmed animals and optimism about further reforms to factory farming.

Exploratory analysis found no evidence that hierarchical meat-eating justification or beliefs about how well-treated farmed animals currently are mediated the effect. Further research is needed to better understand why providing information about animal welfare reforms did not substantially increase AFO overall, whereas providing information about current practice did somewhat increase AFO.

FULL PAPER

Author Accepted Manuscript / Preprint:

Anthrozoös (paywalled):

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Would It Cost to Build a World-Class Dredging Vessel in America?, published by Zvi on May 2, 2022 on LessWrong.

I'm doing some research into questions surrounding the Foreign Dredge Act of 1906, and thought I'd experiment by throwing this out there.

For context, the 31 biggest dredging vessels were not built in America, and thus cannot be used in America by law. We only have a small number of less capable vessels, and they often get redirected to short-term emergency tasks. This is preventing us from doing a bunch of very valuable things, like repairing or expanding ports, which end up taking much more time and money or not happening at all.

This podcast is recommended. You can find a transcript here. They claim that there's no way America will be able to have such capacity for at least decades. I want to verify that (and also check if any other claims here don't ring true)?

As an alternative to repealing the Dredge Act (which I'm exploring and planning to write about) another alternate would be to build world-class dredging vehicles here in America, such that they could be used. Before assuming that this is impossible, and to have a straight answer, what would happen if someone with deep pockets tried to commission a world-class dredging ship that would qualify? Could be done? Are there other impossible barriers to solve? How much would it cost and how much more would that be than building it elsewhere? How long would it take?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing good easier: how to have passive impact, published by Kat Woods on May 2, 2022 on The Effective Altruism Forum.

What’s better - starting an effective charity yourself, or inspiring a friend to leave a low-impact job to start a similarly effective charity? Most EAs would say that the second is better: the charity gets founded, and you’re still free to do other things.

Persuading others to do impactful work is an example of what I call passive impact. In this post, I explain what passive impact is, and why the greatest difference you make may not be through your day-to-day work, but through setting up passively-impactful projects that continue to positively affect the world even when you’ve moved on to other things.

What is passive impact?

When we talk about making money, we can talk about active income and passive income. Active income is money that is linked to work (for example, a salary). Passive income is money that is decoupled from work, money that a person earns with minimal effort. Landlords, for example, earn passive income from their properties: rent comes in monthly and the landlord doesn’t have to do much, beyond occasional maintenance.

Similarly, when we talk about our positive impact, we can talk about active impact and passive impact. When most people think about their impact, they think about what they do. A student might send $100 to the world’s poorest people, who might use this money to buy a roof for their house or education for their kids. Or an AI researcher might spend 2 hours working on a problem in machine learning, to help us make superintelligent AI more likely to share our values. These people are having an active impact - making the world better through their actions. Their impact is active because, in order to have the same impact again, they’d have to repeat the action - make another donation, or spend more time working on the problem.

Now consider the career advisors at 80,000 Hours. Imagine that, thanks to their advice, a young person decides to work for an effective animal advocacy charity rather than at her local cat shelter, and thus save hundreds of thousands of chickens from suffering on factory farms. The 80,000 Hours advisors can claim some of the credit for this impact - after all, without their advice, their advisee would have had a much less impactful career. But after the initial advising session, the coaches don’t need to keep meeting with their advisee - the advisee generates impact on her own. This is what I mean by passive impact: taking individual actions or setting up projects that keep on making the world better, without much further effort.

The ultra-wealthy make most of their money through passive income. Bill Gates hasn’t worked at Microsoft since 2008, but it continues to make money for him. Similarly, many highly successful altruists are most impactful not through their day-to-day work, but through old projects that continue to generate positive impact, without further input.

Why should you try to create passive impact

What are the benefits of passive impact? Here are a few:

You can have a really big impact

Your active impact is limited by your time, energy, and money, but your passive impact is boundless because you can just keep on setting up impactful projects that run in parallel to each other.

It’s satisfying

It’s really pleasing to be lounging on a beach somewhere and to hear that one of my projects has had a positive impact.

It’s more efficient

When I set up the Nonlinear Library, people asked me why I didn’t get a human to read the posts, rather than a machine. But by automating the process, I’m saving loads of money and time. It will take a robot two weeks and $6,000 to record the entire Less Wrong backlog; if we’d hired a human to read all those posts, it would take many years and over a million d...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My thoughts on nanotechnology strategy research as an EA cause area, published by Ben Snodin on May 2, 2022 on The Effective Altruism Forum.

Two-sentence summary: Advanced nanotechnology might arrive in the next couple of decades (my wild guess: there’s a 1-2% chance in the absence of transformative AI) and could have very positive or very negative implications for existential risk. There has been relatively little high-quality thinking on how to make the arrival of advanced nanotechnology go well, and I think there should be more work in this area (very tentatively, I suggest we want 2-3 people spending at least 50% of their time on this by 3 years from now).

Context: This post reflects my current views as someone with a relevant PhD who has thought about this topic on and off for roughly the past 20 months (something like 9 months FTE). Note that some of the framings and definitions provided in this post are quite tentative, in the sense that I’m not at all sure that they will continue to seem like the most useful framings and definitions in the future. Some other parts of this post are also very tentative, and are hopefully appropriately flagged as such.

Key points

I define advanced nanotechnology as any highly advanced future technology, including atomically precise manufacturing (APM), that uses nanoscale machinery to finely image and control processes at the nanoscale, and is capable of mechanically assembling small molecules into a wide range of cheap, high-performance products at a very high rate (note that my definition of advanced nanotechnology is only loosely related to what people tend to mean by the term “nanotechnology”). (more)

If developed, advanced nanotechnology could increase existential risk, for example by making destructive capabilities widely accessible, by allowing the development of weapons that pose a higher existential risk, or by accelerating AI development; or it could decrease existential risk, for example by causing the world’s most destructive weapons to be replaced by weapons that pose a lower existential risk. (more)

Timelines for advanced nanotechnology are extremely uncertain and poorly characterised, but the chance it arrives by 2040 seems non-negligible (I’d guess 1-2%), even in the absence of transformative AI. (more)

It seems likely that there’d be a long period of development with clear warning signs before advanced nanotechnology is developed, pushing against prioritising work in this area and pushing towards a focus on monitoring and foundational work. (more)

There has been relatively little high-quality nanotechnology strategy work, and by default this seems unlikely to change much in the near future. (more)

It seems possible to make progress in this area, for example by clarifying timelines, tracking potential warning signs of accelerating progress, and doing strategic planning. (more)

Overall, I think that nanotechnology strategy research could be very valuable from a longtermist EA perspective. Currently, my extremely rough, unstable guess is that we should have 2-3 people spending at least 50% of their time on this by 3 years from now (against a background of perhaps 0-0.5 FTE over the past 5 years or so). (more)

Note that it seems that we don’t want to accelerate progress towards advanced nanotechnology because of (i) the dramatic but highly uncertain net effects of the technology, including the possibility of very bad outcomes, (ii) the plausible difficulty of reversing an increase in the rate of progress, and (iii) the option of waiting to gain more information. (Though note that I still feel a bit confused about how harmful various forms of accelerating progress might be, and I’d like to think more carefully about this topic.) (more)

Introduction

This post has two main goals:

To provide a resource that EA community...