Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The harms you don't see, published by ViktoriaMalyasova on October 16, 2022 on LessWrong.

Падшие ангелы рядом с лицами старых солдат

Ждут одного лишь приказа: вернуться. Вернуться назад

Чтобы собрать все знамёна, стрелы, обломки мечей,

Вновь пережить вместе битву ста дней и ночей.

Катарсис, "Крылья"I often see people in the West argue that Ukraine ought to cede some territories to Russia so that peace is achieved and people are kept safe. While perharps well intentioned, I think these suggestions are based on a distorted perception of reality. There are harms that you notice, because they happen in a big, loud fight that draws everyone attention, and there are harms that happen not so visibly, in dark overcrowded cells, in dusty courtrooms, in people's heads. You may not hear about them in the news, but it is still important to know and account for them.

In 2014, when Russia annexed Crimea, it started destroying all Crimean Tatar civil society institutions. Mejlis, the executive representative body of Crimean Tatars since 1991, was outlawed as an "extremist organization" in 2016. Russia banned Crimean Tatar national events: the rally on the Deportation of Crimean Tatar People Day on May 18th and the Crimean Tatar National Flag day on June 26th. Over 340 people are being persecuted for their alleged participation in Hizb ut-Tahrir. Over 230 people are already sentenced, and most of them got prison terms of 10 years or more.

Hizb ut-Tahrir is an Islamist political organization that aims to establish Islamic caliphate by recruiting members and advancing them to positions of power. Russia considers them a terrorist organization, which is ridiculous as they never commited or were known to plan any acts of terror. Some countries banned Hizb ut-Tahrir because they have issues with antisemitism and the whole Islamic caliphate thing. But in many European countries, including Ukraine, the organization is legal. Obviously Russia isn't persecuting Hizb ut-Tahrir for their antisemitism, or non-existent terrorism. It is questionable if the convicted people really belong to the organization, all but one of the accused denied it, and one person claimed they were tortured to extract a confession. Russia's real problem with Hizb ut-Tahrir, Mejlis and Crimean Tatars in general is that they are organized groups.

Ukrainians often wonder why are Russian protests so toothless. Why don't we fight for each other, through Molotov cocktails at the cops et cetera. One reason is that the Russian society is very atomized. You sure can start throwing Molotov cocktails on your own. But if other people don't support you, you are just going to land yourself in prison for nothing. In order to protest effectively, you need to coordinate with others. You need to trust that other people will back you up, and they need to trust that you are not an agent provocateur. If you want to use violent tactics, you need to first build common knowledge that we, as a movement, are done with the whole nonviolent resistance thing, it didn't work, and we are trying other methods now. You need to recruit enough people to execute your idea well and run into no undercover cops in the process. You need leaders who people can trust and who aren't secretly working for Kremlin.

This is why Kremlin sees any independent organizations as a threat. Independent media must be destroyed, popular leaders imprisoned and killed, communities disbanded. You must not be able to talk to people, trust people or organize any collective action. Then you cannot resist the government effectively and are left helpless. This is what Russia is doing to people in Crimea and any other territory it controls.

But what happens to people when they are helpless before the government? I could tell you all day about the human right abuses, the widespread to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maximal Lotteries, published by Scott Garrabrant on October 17, 2022 on LessWrong.

Probabilistic Voting Theory

Recall that in the last post I said that a voting system is a function that takes in a distribution on utility functions on a set of candidates, and produces a distribution on that set of candidates, and that voting theorists tend to make four assumptions:

The set of candidates is finite.

The function only uses the preorders on candidates implied by the utility functions.

The output distribution assigns probability 1 to a single candidate.

The input distribution is the uniform distribution on some finite set.

The fourth assumption doesn't really matter, and isn't really used, so we can just remove it. The third assumption, on the other hand, is important for many of the voting theory conclusions, but we are now going to see what happens if we remove it. We will keep the first two assumptions.

So now, we will be considering lotteries (i.e. distributions) over candidates, since that will be the output of our voting system.

Given a set C of candidates, lotteries A,B∈Δ(C), and an electorate V∈Δ(C[0,1]), we say that A dominates B if Pa∼A,b∼B,v∼V(v(a)>v(b))≥Pa∼A,b∼B,v∼V(v(b)>v(a)).

Lets unpack this definition. We are given three distributions V, a distribution on voters, and A and B, distributions on candidates. We are imagining independently sampling from these three distributions to get v, a, and b respectively. When we do this, one of three things will happen:

v will prefer a to b. (v(a)>v(b))

v will prefer b to a. (v(b)>v(a))

v will be indifferent. (v(a)=v(b))

We say that A dominates B if the first outcome is at least as likely as the second outcome. Note that we don't need to look at the actual utility functions to determine whether A dominates B, we only need to look at the partial ordering of preferences over candidates, so we are not violating assumption 2. Also note that domination need not be transitive.

Maximal Lotteries

Given a set C of candidates, and an electorate V∈Δ(C[0,1]), a maximal lottery is an M∈Δ(C) such that for all other L∈Δ(C), M dominates L.

Sounds great, but surely that is too strong a condition, domination isn't even transitive.

Doesn't matter. Maximal lotteries always exist!

What? Is this some silly Brouwer's fixed point thing, where you can't find them, and they aren't unique?

Nope. Maximal lotteries are basically unique. The only reason they aren't unique is because of ties. For example, if your electorate is the uniform distribution on an odd number of voters that are never indifferent, there will be a unique maximal lottery. And you can find them quickly.

Wow, so this is like a proper voting system then: given an electorate, just output the maximal lottery.

Yep.

Is it any good?

Well, it's Condorcet.

Wow! This is has got to be the most elegant Condorcet voting system I have ever seen! Most of the Condorcet systems I have seen have basically looked like they took the Condorcet criterion, and then wrote an algorithm around it. Is it Clone invariant?

Yep.....And it satisfies consistency and participation.

What? Those are incompatible with Condorcet!

Only for deterministic voting systems!

Wow! I don't really have an intuition for probabilistic voting systems, There must be all sorts of cool stuff you can do once you allow randomness.

No, this is basically it. You could choose a random voter to be dictator, which is nice for being strategy-proof, but that isn't Condorcet. In fact, Maximal Lotteries are uniquely characterized as the only probabilistic voting system that is Condorcet, consistent, and clone invariant. (See this paper by Florian Brandl, Felix Brandt, and Hans Georg Seedig.)

I'm sold. But are people ready to start flipping coins in elections?

They usually don't have to! Remember, maximal lotteries satisfy the Condorcet Criter...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Cause Innovation Bootcamp, published by Akhil on October 17, 2022 on The Effective Altruism Forum.

We are pleased to introduce Cause Innovation Bootcamp (CIB), a project that aims to train researchers interested in EA, while vetting new potential cause areas in Global Health and Development. We achieve this by taking research fellows through a training bootcamp that upskills them on the basics of evidence-based research and then getting them to produce a shallow report (using a standardised template) of a cause area, all whilst being supported by a senior mentor. These reports will then be posted on the EA Forum, and be sent to relevant organisations who the research might be of particular interest to, and for whom it might inform their decision-making. Cause areas are selected through a rough prioritisation which helps us identify which ones we think are most likely to be promising. We are currently running a pilot program for 5-6 fellows (7th Nov. 2022- 20th Dec. 2022).

Applications are open until 30th Oct. 2022 (Sunday)- please apply on the following application form if you are interested.

Why this project?

We identified a few primary bottlenecks in the research infrastructure within EA:

A relative lack of novel research exploring new problem areas and interventions that might beat the current bar and/or be as impactful as current EA focus areas within global health and development- We think that this is important because our ability to most effectively do the most good depends on our ability to either fund or found new interventions within cause areas. We try to solve this by having created a comprehensive database of cause areas within global health and global development, and roughly prioritising them based on burden of disease/number of people affected and tractability. By producing shallow reports on topics that might be promising and are currently under-investigated, we hope that this can reveal potential ‘blindspots’ and ‘low-hanging fruit’. Our database and all shallow reports written will be an open-access resource available to the entire EA research community.

A relative lack of quality researchers (both current and emerging) approaching prioritisation from an EA lens- We think, given that quality research is essential to fund or found promising interventions, researchers are pretty central to the long-term health and strength of the EA movement. We aim to mobilise two groups of talent: (1) motivated EAs who want to be involved in research, but don't currently have the requisite skills and experience, (2) existing researchers in the broader GHD (Global Health and Development) community who might be interested in approaching research from an EA perspective.

Although there are lots of fantastic researchers in EA and the broader GHD community, we know that there is a talent bottleneck. We also know that there are lots of motivated people out there who want to be involved in research!

What do we do?

Our theory of change is below

Our current assumption is that roughly 50% of the impact is achieved by introducing more good researchers to the EA research ecosystem. We would estimate that the other 50% of the impact to come from research produced through the training program. We aren’t confident about these estimates, and will evaluate these at the end of our pilot program; depending on their results, we might emphasise parts of this Theory of Change that generate the most impact.

What does being a fellow involve?

We are currently running a pilot of our program, and have received funding from the EA Infrastructure Fund to fund 5- 6 research fellows through our program. The program is a 6-week commitment from November 7 2022 -20 December 2022 that involves:

Self-study of training modules on research skills- We have 6-8 lessons covering various key res...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sign of quality of life in GiveWell’s analyses, published by brb243 on October 16, 2022 on The Effective Altruism Forum.

Content information: This post discusses that lives can be valued negatively by intended beneficiaries.

TLDR: GiveWell can include the sign of life quality variable in its analyses.

Centre for Pesticide Suicide Prevention (CPSP)

In January 2021, GiveWell recommended an approximately $7 million general support grant for the Centre for Pesticide Suicide Prevention (CPSP). The Centre supports the deregistration of pesticides commonly used in suicide.

GiveWell assumes that the Centre accelerates the pesticide deregistration by 9 years in each of the countries of operation. The number of lives saved is calculated as the difference between the numbers of lives saved in different policy scenarios weighted by these scenarios' probabilities and the counterfactual improvement in pesticide suicide rates, as extrapolated from past trends. 25,938 lives are expected to be saved.

The assumption that the intended beneficiaries would otherwise suicide can suggest that they value their lives negatively.

GiveWell assumes that as a result of the pesticide deregistration, agricultural productivity decreases by 30%. The effects of this estimated productivity decrease on the quality of the intended beneficiaries’ lives is not discussed.

CPSP-promoted pesticide bans would affect entire nations with large farming populations, who live in extreme and national poverty. A 30% decrease in yield can result in increased hunger levels and decreased income levels, which can negatively affect the health and quality of life of millions of people.

If the decreases in health and life quality cause a large percentage of farmers to value their lives negatively, support of CPSP can cause millions of people to live dissatisfied.

Thus, while CPSP is expected to save thousands of lives, it can also cause millions to live dissatisfied.

Malaria vaccination in Kenya

The Measuring people’s preferences IDinsight survey shows that the average life satisfaction score of Kenyan respondents is 2.3/10 (n = 1,808, SD = 2.32) (p. 42). If the neutral point, “the point where someone is neither satisfied nor dissatisfied,” is above 2.3/10, reducing mortality in Kenya by malaria vaccination can save lives of dissatisfaction.

A small-scale (n=30) survey that I conducted in early 2021 in a Kenyan slum suggests that many respondents can value their lives negatively. Respondents were asked to label death and their quality of life on a visual analog scale (arrow) from the worst and the best imaginable situation. If worst corresponds to -1, death to 0, and best to 1, the average quality of life was -0.18, with a standard deviation of 0.40 (column AC). On average, respondents wanted to live 13 additional years (SD=26), while the median was 2 additional years.

The wording of the quality of life questions and their survey context can be assessed as leading, which can suggest low quality of data.

If you mai[n]tain your lifestyle and the future is what you expect it to be, how many more years starting from now do you want to live? You can say anything from 0 to forever.

Where do you place death on this arrow? Please put a line there and write "0." . Where do you place yourself on this arrow? Please put a line there and write "myself."

A local enumerator familiar with the area collected the data, while knowing that it is gathered for informational purposes only, without a possible benefit of answering in any specific way. While some (globally poor) respondents accepted a small stipend for their time (which is a custom in the area), many were happy to share their responses without a stipend. This can suggest that respondents answered relatively honestly, with little experimenter bias.

This survey shows that the quality...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My resentful story of becoming a medical miracle, published by Elizabeth on October 16, 2022 on LessWrong.

You know those health books with “miracle cure” in the subtitle? The ones that always start with a preface about a particular patient who was completely hopeless until they tried the supplement/meditation technique/healing crystal that the book is based on? These people always start broken and miserable, unable to work or enjoy life, perhaps even suicidal from the sheer hopelessness of getting their body to stop betraying them. They’ve spent decades trying everything and nothing has worked until their friend makes them see the book’s author, who prescribes the same thing they always prescribe, and the patient immediately stands up and starts dancing because their problem is entirely fixed (more conservative books will say it took two sessions). You know how those are completely unbelievable, because anything that worked that well would go mainstream, so basically the book is starting you off with a shit test to make sure you don’t challenge its bullshit later?

Well 5 months ago I became one of those miraculous stories, except worse, because my doctor didn’t even do it on purpose. This finalized some already fermenting changes in how I view medical interventions and research. Namely: sometimes knowledge doesn’t work and then you have to optimize for luck.

I assure you I’m at least as unhappy about this as you are.

Preface to the Preface

I’ve had nonspecific digestive issues since before I have memories. In pre-school my family joked that I would die as a caveman because there were so few things I would eat, and they were mostly grains. This caused a bunch of subclinical malnutrition issues that took a lot of time to manage and never got completely better. And while I couldn’t articulate this until it went away, food felt gross all the time

It’s hard to convey just how bad this was for me, because it feels like it undermines everything I did to work around it. I’ve always been functional but decidedly less healthy than my friends. I got sick more often and it hit me harder. I was slower to heal from injuries and scrapes and that limited my interest in the more athletic sort of hobbies. I couldn’t work the same hours, and working hours traded off really sharply against energetic hobbies. I had to spend a lot of time managing food where other people can just show up and eat, which was a constant source of social stress. My genetics say I was destined to have anxiety issues, but the low level malnutrition and justified feelings of food insecurity despite apparent abundance did not help anything.

Eventually in my late 20s. I saw a nutrition-focused psychiatrist who listened to my observations (I could only eat protein with soda), immediately formed a hypothesis (I produced insufficient stomach acid), asked questions to rule it out (which I no longer remember), suggested a test (take stomach acid pills and see if they gave me heartburn), and when it came back positive (no heartburn) suggested a course of action (keep taking stomach acid pills) that showed immediate benefits in practice (indigestion removed, but only when I took the pills). My protein and produce intake increased enormously, and I felt overall much better.

This is exactly how I want medicine to work. I gathered good data and took it to an expert who immediately formed a model, definitively tested it, and prescribed a course of action that made mechanistic sense. If you forget that it took almost 30 years and I took those exact same symptoms to other doctors beforehand, it’s a stunning success.

But it was not a total success. My protein intake maxed out at 50 grams/day, and that was if I made consuming protein a hobby and nothing went wrong. I was doing much better than I had been, but my nutrient tests ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Age changes what you care about, published by Dentin on October 16, 2022 on LessWrong.

[Possible trigger warning for discussion of mortality.]

I've been on LessWrong for a very long time. My first exposure to the power of AI was in the mid 1980's, and I was easily at "shock level 4" at the turn of the millennium thanks to various pieces of fiction, newsgroups, and discussion boards.

(If you're curious, "The metamorphosis of prime intellect", "Autonomy", and "A fire upon the deep" all had large impact on me.)

My current best guess is that there's a double-digit percentage chance the human race (uploads included) will go extinct in the next century as a result of changes due to AI.

With that background in mind, existential AI risk is not my highest priority, and I am making no meaningful effort to address that risk other than sometimes posting comments here on LW. This post is an attempt to provide some insight into why, to others who might not understand.

Put bluntly, I have bigger things to worry about than double digit odds of extinction in the next century due to AI, and I am a rather selfish individual.

What could possibly be more important than extinction? At the moment, that would be an extremely solid 50% chance of death in the next 3-4 decades. That's not some vague guesswork based on hypothetical technological advances; it's a hundred thousand people dying per day, every day, as they have been for the last century, each death contributing data to the certainty of that number. That's two decades of watching the health care system be far from adequate, much less efficient.

Consider that right now, in 2022, I am 49 years old. In 2030 I'll be 57, and there's good reason to believe that I'll be rolling 1d100 every year and hoping that I don't get unlucky. By 2040, it's 1d50, and by 2050 it's 1d25. Integrate that over the next three decades and it does not paint a pretty picture.

Allow me a moment to try to convey the horror of what this feels like from the inside: every day, you wonder, "is today the day that my hardware enters catastrophic cascading failure?" Every day, it's getting out a 1d10 and rolling it five times, knowing that if they all come up 1's it's over. I have no ability to make backups of my mind state; I cannot multihost or run multiple replicas; I cannot even export any meaningful amount of data. The high reliability platform I'm currently running on needs to fail exactly once and I am permanently, forever, dead. My best backup plan is a single digit probability that I can be frozen on death and revived in the future. (Alcor provides at best only a few percent odds of recovery, but it's better than zero and it's cheap.)

That's what it feels like after you've had your first few real health scares.

I'd like to say that I'm noble enough, strong enough as a rationalist that I can "do the math" and multiply things out, take into account those billions of lives at risk from AI, and change my answer. But that's where the selfish aspect of my personality comes in: I turns out that I just don't care that much about those lives. I care a lot more about personally surviving the now than to have everyone survive the later.

It's hard to care about the downfall of society when your stomach is empty and your throat is parched. It's hard to care about a century from now, when death is staring you in the face right now.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I learn better when I frame learning as Vengeance for losses incurred through ignorance, and you might too, published by chaosmage on October 15, 2022 on LessWrong.

THE OBSERVATION

My main point is in the title: I have found that when I consciously learn "with a vengeance", aiming to avenge whatever I lost because I have not learnt the thing earlier, I learn markedly better. I feel more motivated to learn, and recall seems clearly better. (I have not formally compared recall before and after, this is subjective judgement but I'm very confident.)

I have experimented with this for about 2 weeks. So far the effect does not seem to diminish; if there is any change at all, it is maybe getting slightly stronger.

Subjectively, this changes my failures from something to mourn or to be embarrassed about, into "something to be avenged" which is a driving motivation. It feels like I'm owning my mistakes more readily. I acknowledge them more readily and am more comfortable thinking and talking about them. And the learning from them feels something like gleeful, justified or gloating - definitely more powerful and satisfying.

I would like some of you to try this, and to report back whether this works for you too, because I have never heard anyone else talk about consciously trying to do this, so it might be new, and it helps me a lot so if it helps others too, it seems potentially extremely useful.

THE NARRATIVE

I was having a great argument about God with a very good friend who is a sophisticated theologian, and who pointed me to a podcast with the German theologian Siegfried Zimmer. Zimmer was talking about theodicy, the classic problem of theism where an almighty, all-knowing, all-good God seems impossible to square with the pointless suffering we observe. He did not have a satisfactory answer, he did manage to admit that, he did not manage to draw the obvious conclusion, so nothing new there. But in his argument he gave a really interesting reason to discard all the usual theistic answers. He said an answer to suffering is only good if you can say it to someone who is intensely and pointlessly suffering, to his or her face, and find it helps.

I was very impressed with this idea and concluded that as an atheist, I did not have an answer that would fulfil this criterion. So I thought about it. What do I say to someone dying in a concentration camp, from the other side of the fence, utterly unable to save them? To a kid dying of cancer? To the grieving parents of victims of a school shooting?

Naturally, as you do, I thought back to the Star Trek parody "Galaxy Quest". The single best scene in the movie is this:

This scene stands out from the rest of the movie because of its utter sincerity. That is the point: the same character has been saying the same line insincerely for many years and is completely sick of it, but he recognises that in this particular situation it is the best thing he could possibly say: "You shall be avenged."

In the concentration camp scenario, if I imagine myself on either side of the fence, I really think that would help. "We can't save you, but we will avenge you." Yeah. That rings good.

And I find this easily generalizes into situations where there isn't a human perpetrator to be punished. The project of eradicating Malaria just feels more viscerally awesome when I frame it as the spiteful, relishing, merciless, victorious extermination of the terrible monster that has, by some estimates, killed around 10% of all humans who have ever lived.

And it works for small things just as well. I paid too much for an item? I avenge the lost money by buying more carefully next time - and I find I actually do remember to do it next time. I lost time because of a scheduling mistake? I avenge it by scheduling better - and my scheduling improves faster than it used to....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A common failure for foxes, published by Rob Bensinger on October 14, 2022 on LessWrong.

A common failure mode for people who pride themselves in being foxes (as opposed to hedgehogs):

Paying more attention to easily-evaluated claims that don't matter much, at the expense of hard-to-evaluate claims that matter a lot.

E.g., maybe there's an RCT that isn't very relevant, but is pretty easily interpreted and is conclusive evidence for some claim. At the same time, maybe there's an informal argument that matters a lot more, but it takes some work to know how much to update on it, and it probably won't be iron-clad evidence regardless.

I think people who think of themselves as being "foxes" often spend too much time thinking about the RCT and not enough time thinking about the informal argument, for a few reasons:

1. A desire for cognitive closure, confidence, and a feeling of "knowing things" — of having authoritative Facts on hand rather than mere Opinions.

A proper Bayesian cares about VOI, and assigns probabilities rather than having separate mental buckets for Facts vs. Opinions. If activity A updates you from 50% to 95% confidence in hypothesis H1, and activity B updates you from 50% to 60% confidence in hypothesis H2, then your assessment of whether to do more A-like activities or more B-like activities going forward should normally depend a lot on how useful it is to know about H1 versus H2.

But real-world humans (even if they think of themselves as aspiring Bayesians) are often uncomfortable with uncertainty. We prefer sharp thresholds, capital-k Knowledge, and a feeling of having solid ground to rest on.

2. Hyperbolic discounting of intellectual progress.

With unambiguous data, you get a fast sense of progress. With fuzzy arguments, you might end up confident after thinking about it a while, or after reading another nine arguments; but it's a long process, with uncertain rewards.

3. Social modesty and a desire to look un-arrogant.

It can feel socially low-risk and pleasantly virtuous to be able to say "Oh, I'm not claiming to have good judgment or to be great at reasoning or anything; I'm just deferring to the obvious clear-cut data, and outside of that, I'm totally uncertain."

Collecting isolated facts increases the pool of authoritative claims you can make, while protecting you from having to stick your neck out and have an Opinion on something that will be harder to convince others of, or one that rests on an implicit claim about your judgment.

But in fact it often is better to make small or uncertain updates about extremely important questions, than to collect lots of high-confidence trivia. It keeps your eye on the ball, where you can keep building up confidence over time; and it helps build reasoning skill.

High-confidence trivia also often poses a risk: either consciously or unconsciously, you can end up updating about the More Important Questions you really care about, because you're spending all your time thinking about trivia.

Even if you verbally acknowledge that updating from the superficially-related RCT to the question-that-actually-matters would be a non sequitur, there's still a temptation to substitute the one question for the other. Because it's still the Important Question that you actually care about.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The US expands restrictions on AI exports to China. What are the x-risk effects?, published by Stephen Clare on October 14, 2022 on The Effective Altruism Forum.

Late last week, the Biden administration announced a new set of regulations that make it illegal for US companies to export a range of AI-related products and services to China (Financial Times coverage here (paywalled 👀); Reuters here). By all accounts this new export controls policy is a big deal for US-China competition and Chinese AI progress. Its wide-reaching effects touch on a number of issues important to effective altruists, AI safety strategists, and folks interested in reducing global catastrophic risks. This is a quickly-written post in which I summarize my understanding of the new regulations and their probable effects, then list some questions the announcement prompted regarding the effect on various global catastrophic risks.

The new export controls policy

If you pick a random article commenting on the new export controls policy, you’ll probably see some dramatic language. Reuters describes the new rules as an attempt to “hobble” China’s chip industry. CSIS’s Greg Allen writes that they seek to “[choke] off China’s access to the future of AI”. ChinaTalk’s Jordan Schneider thinks they “will reshape the global semiconductor industry and the future of the US-China relationship”.

The export controls seek to slow or stop Chinese companies from developing cutting-edge AI capabilities. They do this by making it illegal for US companies, US citizens, and foreign companies that use US products to sell AI-related hardware, software, and services to Chinese entities. This kind of policy is not new in itself. For years, the US has prohibited companies from selling AI tech to China for military use. However, the new regulations are notable for being much broader than previous restrictions and broad-based, no longer restricted to military uses in particular.

Allen writes that the regulations target four “chokepoints” in the AI development supply chain. Restrictions have been placed on the export of high-end computer chips, chip design software, chip manufacturing equipment, and manufacturing equipment components. I am going to summarize Allen’s explanations very briefly. If this is relevant to your work I strongly recommend you go read his article, which is packed with interesting and important details. Schneider also has a digestible tweet thread explanation you can read here.

1. The best computer chips

The new policies will functionally end the sale of the “best” computer chips, i.e. any chip above a certain performance threshold, to China. The US had already blocked the sale of such chips to China’s military. However, Allen writes that this policy was ineffective due to military-civil fusion in China; the boundary between military and non-military organizations in China is purposefully and officially blurry. Unable to distinguish military from non-military uses, the US government has decided to just block the sale of high-end chips to China entirely.

2. The software used to design chips

Advanced chips are so complex that specialized software is required to design them. The new regulations also ban companies from providing this software to Chinese companies. This will make it harder for Chinese companies to design new chips that can compete with the high-end chips produced by, e.g., Nvidia and AMC, to which they have just lost access.

3. The equipment used to make semiconductors

Allen writes that Chinese companies could yet circumvent chokepoints (1) and (2) by designing and manufacturing chips using older programs and equipment. But chokepoint (3) seeks to stop this by banning the sale of chip manufacturing equipment that exceeds a certain performance threshold. US citizens are also no longer allowed to ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counterarguments to the basic AI risk case, published by Katja Grace on October 14, 2022 on The Effective Altruism Forum.

This is cross-posted from the AI Impacts blog

This is going to be a list of holes I see in the basic argument for existential risk from superhuman AI systems1.

To start, here’s an outline of what I take to be the basic case2:

I. If superhuman AI systems are built, any given system is likely to be ‘goal-directed’

Reasons to expect this:

Goal-directed behavior is likely to be valuable, e.g. economically.

Goal-directed entities may tend to arise from machine learning training processes not intending to create them (at least via the methods that are likely to be used).

‘Coherence arguments’ may imply that systems with some goal-directedness will become more strongly goal-directed over time.

II. If goal-directed superhuman AI systems are built, their desired outcomes will probably be about as bad as an empty universe by human lights

Reasons to expect this:

Finding useful goals that aren’t extinction-level bad appears to be hard: we don’t have a way to usefully point at human goals, and divergences from human goals seem likely to produce goals that are in intense conflict with human goals, due to a) most goals producing convergent incentives for controlling everything, and b) value being ‘fragile’, such that an entity with ‘similar’ values will generally create a future of virtually no value.

Finding goals that are extinction-level bad and temporarily useful appears to be easy: for example, advanced AI with the sole objective ‘maximize company revenue’ might profit said company for a time before gathering the influence and wherewithal to pursue the goal in ways that blatantly harm society.

Even if humanity found acceptable goals, giving a powerful AI system any specific goals appears to be hard. We don’t know of any procedure to do it, and we have theoretical reasons to expect that AI systems produced through machine learning training will generally end up with goals other than those they were trained according to. Randomly aberrant goals resulting are probably extinction-level bad for reasons described in II.1 above.

III. If most goal-directed superhuman AI systems have bad goals, the future will very likely be bad

That is, a set of ill-motivated goal-directed superhuman AI systems, of a scale likely to occur, would be capable of taking control over the future from humans. This is supported by at least one of the following being true:

Superhuman AI would destroy humanity rapidly. This may be via ultra-powerful capabilities at e.g. technology design and strategic scheming, or through gaining such powers in an ‘intelligence explosion‘ (self-improvement cycle). Either of those things may happen either through exceptional heights of intelligence being reached or through highly destructive ideas being available to minds only mildly beyond our own.

Superhuman AI would gradually come to control the future via accruing power and resources. Power and resources would be more available to the AI system(s) than to humans on average, because of the AI having far greater intelligence.

Below is a list of gaps in the above, as I see it, and counterarguments. A ‘gap’ is not necessarily unfillable, and may have been filled in any of the countless writings on this topic that I haven’t read. I might even think that a given one can probably be filled. I just don’t know what goes in it.

This blog post is an attempt to run various arguments by you all on the way to making pages on AI Impacts about arguments for AI risk and corresponding counterarguments. At some point in that process I hope to also read others’ arguments, but this is not that day. So what you have here is a bunch of arguments that occur to me, not an exhaustive literature review.

Counterarguments

A. Contra “...