Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Embedding safety in ML development, published by Chin Ze Shen on October 31, 2022 on The AI Alignment Forum.

This post was written as part of Refine. Thanks to Adam Shimi, Alexander Gietelink Oldenziel, and Vanessa Kosoy for helpful discussion and feedback.

Summary

This post aims to:

Advocate for embedding safety into development of machine learning models

Propose a framing on how to think about safety, where three factors contribute to an AI being dangerous, namely misalignment, optimization, and influence

Discuss the pros and cons of this framing

None of the ideas in this post are novel, and some of them may not even be practical, but they may be a useful framing to the problem of AI safety.

Introduction

Motivations

There are many ways to get to AGI. It could be recursively self improving AI, or something more brain-like, or something that comes from improvements in machine learning systems, or something else much more alien to us. I don’t know which pathways are the ones our world will eventually end up in, but given recent rapid advances in ML recently, in this post I will address the scenario where the first dangerous AGI is achieved by mostly scaling up and improving upon present-day ML systems.

In the absence of a robust solution to the hard problem of alignment, I think we should at least try to make the development of machine learning models go safely for as long as possible. This may be especially valuable if the first AGI we get is not yet an all-powerful superintelligent. It might be a fairly weak AGI capable of inflicting a lot of damage, but the damage could have been prevented. Not having a solution to the hard problem should not mean getting ourselves into preventable disasters.

What safety means in this context

Most safety fields in the real world don’t solve safety with guarantees. Whether it is flying an aircraft, operating a nuclear plant, or constructing a building, safety is all about effective risk management. This framing of safety has not been commonly used in the field of AI safety, for very good reasons, as the problem of a misaligned AGI is an unbounded one, where a ‘safety failure’ with a strong form of AGI could mean human extinction even with just a tiny degree of misalignment.

For the purpose of this post, I am using the word “safety” in a more narrow sense than most people would. In terms of time-scale, I refer to “we are safe for now” rather than “things go great in the long run”; while in terms of failure mode, I refer to “we are safe from misaligned AIs” rather than “we are safe from AIs being used by bad actors and multipolar failures etc”.

Framing the problem

The question to be addressed is: what makes an AI harmful? I propose a framing that contains three ingredients:

Misalignment at any level

Optimization towards some goal

Influence over the world

The combination of optimization and influence can be loosely thought of as agency. Without any of the above ingredients, there will not be a catastrophe caused by a misaligned ‘agentic’ AI, as there would only be the following kinds of AI:

Aligned: A fully aligned AI that has strong optimization and high influence over the world will do exactly what we want effectively, possibly leading us to utopia.

Dormant: A misaligned AI that has a lot of influence over the world but not trying to do anything will probably not really be doing anything, like a deactivated nuclear bomb.

Sponge: A misaligned AI that has a lot of optimization power but no influence over the world will not be doing anything in the real world, effectively being a sponge AI. I consider most of the powerful AIs today, such as AlphaGo Zero and DALL-E 2, as sponges for now, as they neither construct world-models nor have physical control over things in the physical world, but it is debatable if they will remain so in the fu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing EA Survey 2022, published by David Moss on October 31, 2022 on The Effective Altruism Forum.

The 2022 EA Survey is now live at the following link:

We appreciate it when EAs share the survey with others. If you would like to do so, please use this link () so that we can track where our sample is recruited from.

We currently plan to leave the survey open until December the 1st, though it’s possible we might extend the window, as we did last year.

What’s new this year?

The EA Survey is substantially shorter. Our testers completed the survey in 10 minutes or less.

We worked with CEA to make it possible for some of your answers to be pre-filled with your previous responses, to save you even more time. At present, this is only possible if you took the 2020 EA Survey and shared your data with CEA. This is because your responses are identified using your EffectiveAltruism.org log-in. In future years, we may be able to email you a custom link which would allow you to pre-fill, or simply not be shown, certain questions which you have answered before, whether or not you share your data with CEA, and there is an option to opt-in to this in this year’s survey.

Why take the EA Survey?

The EA Survey provides valuable information about the EA community and how it is changing over time. Every year the survey is used to inform the decisions of a number of different EA orgs. And, despite the survey being much shorter this year, this year we have included requests from a wider variety of decision-makers than ever before.

Prize

This year the Centre for Effective Altruism has, again, generously donated a prize of $1000 USD that will be awarded to a randomly selected respondent to the EA Survey, for them to donate to any of the organizations listed on EA Funds. Please note that to be eligible, you need to provide a valid e-mail address so that we can contact you.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: love, not competition, published by carado on October 30, 2022 on LessWrong.

many people are bemoaning that AI is going to replace them. this includes notably artists, but we can expect it to start covering mathematicians and, as AI advances, eventually every kind of human endeavor.

there are real, important material concerns, such as artists losing their income, or AI getting so powerful that it destroys everything. this post is not about that, but rather about the longer-term concern of ethically grounding the value of art.

is it okay that AI is outcompeting our creativity? yes! in my opinion, we should never have been grounding valuing ourselves in our ability to be the best at stuff to begin with. we should love ourselves and what we make and do intrinsically, not instrumentally.

it is valid to want to just watch art for the pleasure that that gives you, and it's even okay to wirehead yourself. but it's also valid to value art as a form of communication between real persons, as a special case of the fact that it's valid to care about reality, even if you can't tell.

and the fact that we currently can't tell if art was made by persons or AIs is only a temporary issue; with properly aligned AI, we should be able to tell it "i only want art made by humans!" and have it ensure we only get that, whatever that request would mean upon sufficient reflection.

artists, mathematicians, philosophers, and humans in general: aim not to compete! i, and no doubt many others, value you and the things you make for the fact that they are yours and you are real, in a way that fundamentally, intrinsically excludes purely AI-made art, and which includes art made with a mixture of human and AI work in whatever way i would eventually find reasonable if i thought about it enough.

if you want to just love doing things and love things others have done, you can just do that.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantifying the impact of grantmaking career paths, published by Joel Becker on October 30, 2022 on The Effective Altruism Forum.

How impactful is it to try to become a grantmaker focused on pressing world problems? Encouraged by Nuno’s tweet, I spent a couple of hours trying to find out.

TL;DR

I estimate the expected x-risk basis points reduced for the prospective marginal grantmaker to be 0.12 (with enormous error bars and caveats).

Background

Epistemic status: I have zero grantmaking experience and haven’t even double-checked my numbers; please do not use this post for anything that relies on it not containing serious errors.

My estimand is something like: “expected x-risk basis points reduced for the prospective marginal grantmaker doing their best to benefit the most pressing problems at an Open Philanthropy-equivalent organization.”

BOTEC

Chance of getting a job

I make guesses of

the number of employers that might be possible to work at,

the Open Philanthropy-equivalentness of employers, and

the probability of acceptance for any particular job application.

Multiplying these factors together gives me a probability of succeeding in getting an Open Philanthropy-equivalent grantmaking job offer at (at least) one of the Open Philanthropy-equivalent organizations of 7% (on average). Seems not-crazy.

Impact from grantmaking

I used recent data from these charts to guess the total longtermist grants given per year by Open Philanthropy-equivalent organizations going forwards.

The fraction of grants a prospective grantmaker is responsible for is given by the reciprocal of my guess of the number of grantmaker-equivalents at Open Philanthropy-equivalent organizations. (I apologize for phrasing...)

Only a fraction of the grants that a grantmaker is responsible for can be counterfactually attributed to them — some grants would make for such obvious decisions that the counterfactual grantmaker would have acted identically. I convert the vibes from this post into a fraction of grants worth deliberating over (on average, 13%).

I set the ratio of skill vs. a replacement grantmaker to be mean 1.2 with mass either side of 1. This reflects the fact grantmakers may be better or worse than their counterfactual replacement, but also that on average their being hired implies that they are expected to perform better.

Taken together, I get a mean estimate of $5.7m for the Open Philanthropy-equivalent resources counterfactually moved by grantmaking activities of the prospective marginal grantmaker, conditional on job offer.

Impact from other paths

Linch claims that some fraction of grantmakers' impact comes from things that look like ‘improving all grantmaking processes at their organization,’ ‘improving the work/selection of their grantees,’ and other paths that I did not attempt to categorize. I add the categorized paths in to my estimate of impact.

My mean estimate for the Open Philanthropy-equivalent impact from non-grantmaking activities sums to around $30m (equivalent). ~70% of this comes from improving all grantmaking, ~30% from improving grantees.

Units

In order to calculate impact for the whole career path, I multiply annual impact by the number of years employed as a grantmaker.

Finally, to convert impact into terms of x-risk basis points, I divide by USD per x-risk basis point. Estimates for this quantity come from this comment.

Putting everything together

All considered, I get an estimate of the x-risk basis points reduced by the prospective marginal grantmaker of 0.12 in expectation. There is a ~27% chance of having a negative counterfactual impact (vs. replacement grantmaker) and a ~5% chance of having an impact greater than expected.

For full details, see my code in the appendix.

Limitations

There might be over- or under-counting issues. For instance, I guess that the averag...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longtermist terminology has biasing assumptions, published by Arepo on October 30, 2022 on The Effective Altruism Forum.

Sequence summary

This sequence investigates the expected loss of value from non-extinction global catastrophes. This post is a criticism of the biases and ambiguities inherent in longtermist terminology (including ‘global catastrophes’). The next post, A proposed hierarchy of longtermist concepts, lays out the terms which I intend to use for the rest of this sequence, and which encourage less heuristic, more expected-value thinking. Finally, for now, Modelling civilisation after a catastrophe lays out the structure of a proposed model which will inform the direction of my research for the next few months. If feedback on the structure is good, later parts will populate the model with some best-guess values, and present it in an editable form.

Introduction

Longtermist terminology has evolved haphazardly, so that much of it is misleading or noncomplementary. Michael Aird wrote a helpful post attempting to resolve inconsistencies in our usage, but that post’s necessity and its use of partially overlapping Venn diagrams - implying no formal relationships between the terms - itself highlights these problems. Moreover, during the evolution of longtermism, assumptions that originally started out as heuristics seem to have become locked in to the discussion via the terminology, biasing us towards those heuristics and away from expected value analyses.

In this post I discuss these concerns, but since I expect it to be relatively controversial and it isn’t really a prerequisite for the rest of the sequence so much as an explanation of why I’m not using standard terms, I would emphasise that this is strictly optional reading for the rest of the sequence, so think of it as a 'part 0' of the sequence. You should feel free to skip ahead if you disagree strongly or just aren’t particularly interested in a terminology discussion.

Concepts under the microscope

Existential catastrophe

Recreating Ord and Aird’s diagrams of the anatomy of an existential catastrophe here, we can see an ‘existential catastrophe’ has various possible modes:

Figure from The Precipice

Venn diagram figures all from Aird’s post

It’s the ‘failed continuation’ branch which I think needlessly muddies the waters.

An ‘existential catastrophe’ doesn’t necessarily relate to existence.

In theory an existential catastrophe can describe a scenario in which civilisation lasts until the end of the universe, but has much less net welfare than we imagine it could have had.

This seems odd to consider an ‘existential’ risk - there are many ways in which we can imagine positive or negative changes to expected future quality of life (see for example Beckstead’s idea of trajectory change). Classing low-value-but-interstellar outcomes as existential catastrophes seems unhelpful both since it introduces definitional ambiguity over how much net welfare must be lost for them to qualify, and since questions of expected future quality of life are very distinct from questions of future quantity of life, and so seem like they should be asked separately.

. nor involve a catastrophe that anyone alive recognises

The concept also encompasses a civilisation that lives happily on Earth until the sun dies, perhaps even finding a way to survive that, but never spreading out across the universe. This means that, for example, universal adoption of a non-totalising population ethic would be an existential catastrophe. I’m strongly in favour of totalising population ethics, but this seems needlessly biasing.

‘Unrecoverable’ or ‘permanent’ states are a superfluous concept

In the diagram above, Ord categorises ‘unrecoverable dystopias’ as a type of existential risk. He actually seems to consider them necessarily impermanent, but (in...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Normal" is the equilibrium state of past optimization processes, published by Alex Altair on October 30, 2022 on LessWrong.

Orienting around the ideas and conclusions involved with AI x-risk can be very difficult. The future possibilities can feel extreme and far-mode, even when we whole-heartedly affirm their plausibility.

It helps me to remember that everything around me that feels normal and stable is itself the result of an optimization process that was, at the time, an outrageous black swan.

Modernity

If you were teleported into the body of a random human throughout history, then most likely, your life would look nothing like the present. You would likely be a hunter-gatherer, or perhaps a farmer. You would be poor by any reasonable standard. You would probably die as a child. You would have nothing resembling your current level of comfort, and your modern daily life would be utterly alien to most humans. What currently feels normal is a freeze-frame state of a shriekingly fast feedback loop involving knowledge, industry, and population. It is nowhere near normal, and it is nowhere near equilibrium.

The Anthropocene

For hundreds of millions of years, the earth was a wilderness, teeming with life. The teeming had ebbs and flow, evolutionary breakthroughs and power struggles, but essentially, it was a type of equilibrium.

For hundreds of thousands of years, humans were just another animal. We ran around the savanna, killed other animals and foraged for food, had sex, slept, and experienced joys and losses. If an alien were gazing at earth from afar, they would have had no reason to think humans were different from any other animal.

But all the while, humans were noticing things. They were curious, and desperate, and they had a capacity that the other animals didn't have. They got ideas about the world, and tested them with their hands, and gave this knowledge to their children with their words. Over time, the humans became conspicuous, and eventually, they transformed every inch of their surroundings.

For a while now, humans have been a threat to the species they live with. If you are a fish or a weed or a polar bear, your biggest problem is humans. But as John Green points out in The Anthropocene Reviewed, in the 21st century, if you are the atmosphere or a river or a desert, your biggest problem is humans. The rise of humanity is no longer just an ecological phenomenon; it has kicked off a new geological epoch. The fate of the rock itself is in our hands, in a way that it never was for the other mammals, or the dinosaurs, or the trilobites.

The invention of wood

One day, plants invented wood. This was great for plants, but it had externalities. As wikipedia puts it;

The evolution of the wood fiber lignin and the bark-sealing, waxy substance suberin variously opposed decay organisms so effectively that dead materials accumulated long enough to fossilise on a large scale.

I don't know exactly how this went down. But I like to imagine watching a forest grow on fast-forward; trees spring up, growing taller and taller, and then eventually each tree dies. Only, when these proto-trees falls down, the logs does not sink into the forest floor and dissolve into biodegraded soil. They just stay there. The tree trunks pile up for millions of years. This was the new equilibrium. The plant matter went through some chemical changes that compressed and homogenized it, but much of the plant's stored energy remained. The energy sank deep underground as what we now call fossil fuels.

Bacteria and fungi continued evolving, and eventually they figured out how to eat the now-abundant tougher plant matter. But the coal deposits were now beyond their reach.

The historical details of these transitions are not well understood, but what is clear is that something wild happened, the surfa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Every.org Donation Matching for Monthly Donations, published by Cullen OKeefe on October 30, 2022 on The Effective Altruism Forum.

From Nov. 1 - Nov. 30, or until funds run out, any donor who sets up a monthly donation to your nonprofit will be matched up to $50.00 of their monthly donation for the first two months. This must be a new monthly donation and not one that is currently active. This recurring donation will be automatically charged to the donor's payment method every month and matched another $50.00 for the month of December as well. In total, donors giving $100 ($50 in November and $50 in December) will have $100 matched ($50 in November and $50 in December). Donors who set up a monthly gift in December are not eligible for this match.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Map of Biosecurity Interventions, published by James Lin on October 30, 2022 on The Effective Altruism Forum.

Summary

I recently created a visual map of biosecurity. It covers my interpretation of foundational biosecurity interventions. The map is designed to broadly answer “what are people working on in biosecurity?”

(Link to a larger, higher resolution image)

‘Low-downside Interventions’ are interventions that are on the lower-risk side. These might be good discussion topics or companies to work on. Promising GCBR Interventions are interventions with general consensus among biosecurity experts for being effective at preventing catastrophic pandemics.

Not all interventions are made equal. Some are much more effective than others, but I’ve included the ‘less-promising’ interventions all the same. I didn’t want my biases (or the biosecurity community’s biases, for that matter) about what is and isn’t promising to obscure solutions from a first-principles breakdown.

Though I consulted experts and peers, all final decisions were ultimately mine and may differ from the biosecurity community. The rest of the post will go over my rationale for certain decisions.

GCBR vs. Covid-scale pandemics

This map covers preventing pandemics, not just the catastrophic ones, but also the Covid-scale ones which might occur more frequently.

EA focuses a lot on global catastrophic biological risk (GCBR) pandemics. This seems pretty important given its neglectedness and potentially catastrophic impact. I’ve outlined the interventions that I think would be promising for GCBRs.

At the same time, many in biosecurity consider the effects of AI to outweigh those from biorisks (the magnitude is usually 10-100x). Progress in alignment is influenced by factors like international cooperation, institutional distrust, and nuclear conflict. All of these factors are affected by Covid-scale pandemics, whether through decreases in trust, sudden scarcity of resources, or armed international conflict. Considering the possible detrimental effect of even small-scale but more frequent pandemics, it seems worth exploring interventions focused on this scale. There may be more low-hanging fruit than we imagine.

Mitigate-Prevent breakdown

There are other excellent frameworks for pandemic prevention. Kevin Esvelt’s Delay-Detect-Defend is one such example. I decided on a two-factor framework of Mitigate-Prevent because I think it covers the space of interventions well and because it resulted in a nice visual. Part of this framework is Survive which involves bunkers and food resilience. I didn’t include it because I wanted more emphasis on preventing diseases from becoming pandemics in the first place, which is much more important than escaping to a bunker. If we’re escaping to bunkers, then it’s likely that we’ve lost. That said, I could imagine creating another map that covers this axis of resilience.

Thoughts on some specific interventions

Agricultural disease monitoring and farm-animal testing

Factory farms practically incubate viruses due to the high concentration of animals. Because diseases can pass from animals to humans, more vigilance here could be an effective preventative measure. Wet markets experienced pushback for being viral breeding grounds, and it’s about time that factory farms undergo the same scrutiny.

In addition to worrying about human and animal infection, crops are also vulnerable. Agricultural practices are monoculture in nature, meaning there is a lack of genetic diversity in the crops grown. Blights exist, and countries are surprisingly dependent on a few sources of food. Monitoring agricultural diseases through runoff water or random sampling seems worth consideration.

Bioenhancement

By bioenhancement, I mean health and performance-boosting drugs. Most people have a negative connotation w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental ignoring AI, Dumb but not useless., published by Donald Hobson on October 30, 2022 on The AI Alignment Forum.

This post is yet another unworkable agent design that uses infinite compute. It outlines an AI that ignores all instrumental values, pursuing only it's terminal values.

What is does it look like to pursue an instrumental value. It means you can predict the AI is likely to do action X, even if you have no idea what it's utility function is.

This AI doesn't do that. It obeys the simple principle that, if you are clueless about it's utility function, then you are clueless about it's actions.

Let ΔX mean the probability distribution over the set X.

Lets have some set of Observations O, Actions A and (bounded? ) Utilities U.

Then Pick some probability distribution over utility functions U∈ΔU, such as uniform or complexity weighted. And some probability distribution A∈ΔA over the action space.

Then let the AI be a function f:(U,O)ΔA

We can formalize our condition as ∀o∈O:∀a∈A:∑u∈UP(f(u,o)=a)U(u)=A(a).

(Note U(u)∈R refers to the probability that U assigns to u.)

Why might we expect such agents to be safe. Because if we picked a random utility function, we get a safely random output. So all humans need to do is be better than random in our choice of utility function.

Now let E(o,a,u) be the expected utility. Choose f to maximize ∑u∈U∑a∈AP(f(u,o)=a)U(u)E(u,o,a) subject to the condition. As the condition doesn't relate cases with different values of o, this optimization only needs to be computed for the particular o∈O observed.

Is this AI design safe when repeatedly called with the same utility function?

Imagine the action space just consists of outputting 0 or 1. And there are only 2 utility functions, paperclips and staples. We could imagine that when run millions of times with a utility of paperclips, this AI outputs code for a paperclip maximizer, and when run millions of times with a goal of staples, it outputs a bitwise negation of the paperclip maximizer. (Which could possibly be a staple maximizer, if the AI has done some op code magic)

This would be because the humans choice of the same utility function each time is worse than random. I think this is an actual problem. At least if the AI's have total knowledge of past and future, then this is one Schelling point to their coordination game. (The consider all possibilities doesn't quite work well with multiple different AI's)

I have an intuition that something TDT'ish might fix this.

The other problem that comes to mind is that it isn't easy to whitelist a limited set of instrumental actions you are ok with. You can fake it though, suppose you have a robot, instead of piping the output of this AI directly into the motors, you have it call simple hardcoded functions to walk and pick up objects. The result being, if the AI has a random utility function, instead of thrashing randomly on the floor, it walks to random places and picks up random objects. Because someone hardcoded those low level commands into the robot.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Request for input: how has 80,000 Hours influenced the direction of the EA community over the past 2 years?, published by Ardenlk on October 29, 2022 on The Effective Altruism Forum.

I'm currently helping assess 80,000 Hours' impact over the past 2 years.

One part of our impact is ways we influence the direction of the EA community.

By "direction of the EA community," I mean a variety of things like:

What messages seem prevalent in the community

What ideas gain prominence or become less prominent

What community members are interested in

To get a better understanding of this, I'm gathering thoughts from community members on how they perceive 80,000 Hours to have influenced the direction of the EA community, if at all.

If you have ideas, please share them using this short form!

Note we are interested to hear about both positive and negative influences.

This isn’t supposed to be a rigorous or thorough survey -- but we think we should have a low bar for rapid-fire surveys of people in the community that could be helpful for giving us ideas or things to investigate.

Thank you! Arden

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.