The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Want to make a difference on policy and governance? Become an expert in something specific and boring, published by ASB on August 31, 2023 on The Effective Altruism Forum.

I sometimes get a vibe that many people trying to ambitiously do good in the world (including EAs) are misguided about what doing successful policy/governance work looks like. An exaggerated caricature would be activities like: dreaming up novel UN structures, spending time in abstract game theory and 'strategy spirals', and sweeping analysis of historical case studies.

Instead, people that want to make the world safer with policy/governance should become experts on very specific and boring topics. One of the most successful people I've met in biosecurity got their start by getting really good at analyzing obscure government budgets.

Here are some crowdsourced example areas I would love to see more people become experts in:

Legal liability - obviously relevant to biosecurity and AI safety, and I'm especially interested in how liability law would handle spreading infohazards (e.g. if a bio lab publishes a virus sequence that is then used for bioterrorism, or if an LLM is used maliciously in a similar way).

Privacy / data protection laws - could be an important lever for regulating dangerous technologies.

Executive powers for regulation - what can and can't the executive actually do to get AI labs to adhere to voluntary security standards, or get DNA synthesis appropriately monitored?

Large, regularly reauthorized bills (e.g., NDAA, PAHPA, IAA) and ways in which they could be bolstered for biosecurity and AI safety (both in terms of content and process).

How companies validate customers, e.g., for export control or FSAP reasons (know-your-customer), and the statutes and technologies around this.

How are legal restrictions on possessing or creating certain materials justified/implemented e.g. Chemical Weapons Convention, narcotics, Toxic Substances Control Act?

The efficacy of tamper-proof and tamper-evident technology (e.g. in voting machines, anti-counterfeiting printers)

Biochemical supply chains - which countries make which reagents, and how are they affected by export controls and other trade policies?

Consumer protection laws and their application to emerging tech risks (e.g. how do product recalls work? Could they apply to benchtop DNA synthesizers or LLMs?)

Patent law - can companies patent dangerous technology in order to prevent others from developing or misusing it?

How do regulations on 3d-printed firearms work?

The specifics of congressional appropriations, federal funding, and procurement: what sorts of things does the government purchase, how does this relate to biotech or AI (software)? Related to this, becoming an expert on the Strategic National Stockpile and understanding the mechanisms of how a vendor managed inventory could work.

A few caveats. First, I spent like 30 minutes writing this list (and crowdsourced heavily from others). Some of these topics are going to be dead ends. Still, I'd be more excited about somebody pursuing one of these concrete, specific dead ends and getting real feedback from the world (and then pivoting), rather than trying to do broad strategy work and risk ending up in a never-ending strategy spiral. Moreover, the most impactful topics are probably not on this list and will be discovered by somebody who got deep into the weeds of something obscure.

For those of you that are trying to do good with an EA mindset, this also means getting out of the EA bubble and spending lots of time with established experts in these relevant fields. Every so often, I'll get the chance to collect biosecurity ideas and send them to interested people in DC. In order to be helpful, these ideas need to be super specific, e.g. this specific agency needs to task this other subag...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More than Earth Warriors: The Diverse Roles of Geoscientists in Effective Altruism, published by chanakin on August 31, 2023 on The Effective Altruism Forum.

A global community of and for Effective Geoscientists

tl;dr

We, a small group of geoscientists want to gather a supportive community for knowledge sharing, collaboration, and placing geoscientists EAs in high impact roles.

The use of geospatial data and spatial analysis is underrated. Four out of eight top charities from GiveWell have used geospatial data as part of their research. Particularly, we can have a significant impact in catastrophe resilience.

We would like to advocate for more researchers and effective charities to consider including a geospatial dimension when conducting their studies.

Currently, we will gather on channel role-geoscientists on EA Anywhere slack. Please join us if you are interested.

We listed a few distilled topics that geoscientists EA might be interested in.

Acknowledgement

The completion of this post would not have been possible without the extensive insight, advice, and knowledge shared by the following individuals: Prof. David Denkenberger, Ewelina Hornicka, Petya Kangalova, Leon Mayer, Dr. Kajetan Chrapkiewicz. Any mistakes or oversights in this post are solely my responsibility.

Context and reason for establishment

Figure 1. Both expected impact and personal fit in our career choice reflect a power law.

We want to gather an online community to bring together geoscientists of diverse disciplines. There are two motivations for starting this community. Firstly, we believe that geoscientists and geospatial data can offer value in multiple EA cause areas and intercause research. Secondly, we found that services of current EA infrastructure (e.g. High Impact Professional) did not adequately address the need for our niche speciality and comparative advantage of geoscientists. We want to inform the EA world that we geoscientists exist and our skills and domain expertise can contribute to high impact cause areas. Furthermore, we would like to advocate for more researchers to consider including a geospatial dimension when conducting their studies. Therefore, we would like to bring together geoscientists, current and future, that work in environmental anthropology, epidemiology, remote sensing, geophysics and more.

We want to use this community to highlight existing geospatial research happening in the various cause areas, to support each other in interdisciplinary collaboration and place each other in high impact roles.

What is geospatial data?

We use geospatial data everyday, to track our movement, plan our commute. Geospatial data is any information with an addition of locality attached to it, this is usually represented by Longitude and Latitude. Geospatial data can come from many sources, from the satellite images and radar backscatter to submarine bathymetric surveys.

Figure 2. Alexander von Humboldt's "A Portrait of Nature" in his famous book Der Kosmos (1849).

The development of geospatial science and geospatial data emerged out of necessity. Cartography, the making of maps is the central activity of collecting and distilling geographical knowledge. Initially we made maps for navigation and exploration, then the map became a tool for managing communities, resources, and people, often involuntarily. The map room eventually became the centre of European colonialism in the Age of Discovery (Cresswell T., 2013), and the map room has remained a mainstay of today's military units' control and command. Nowadays, geospatial data are used broadly by ecologists and palaeontologists, by sailors and urban planners, by militaries and humanitarian workers alike. While people talk about the temporal component often, the spatial component often goes unnoticed. This post will attempt to demonstr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #27: Portents of Gemini, published by Zvi on August 31, 2023 on LessWrong.

By all reports, and as one would expect, Google's Gemini looks to be substantially superior to GPT-4. We now have more details on that, and also word that Google plans to deploy it in December, Manifold gives it 82% to happen this year and similar probability of being superior to GPT-4 on release.

I indeed expect this to happen on both counts. This is not too long from now, but also this is AI #27 and Bard still sucks, Google has been taking its sweet time getting its act together. So now we have both the UK Summit and Gemini coming up within a few months, as well as major acceleration of chip shipments. If you are preparing to try and impact how things go, now might be a good time to get ready and keep your powder dry. If you are looking to build cool new AI tech and capture mundane utility, be prepared on that front as well.

Table of Contents

Introduction.

Table of Contents. Bold sections seem most relatively important this week.

Language Models Offer Mundane Utility. Summarize, take a class, add it all up.

Language Models Don't Offer Mundane Utility. Not reliably or robustly, anyway.

GPT-4 Real This Time. History will never forget the name, Enterprise.

Fun With Image Generation. Watermarks and a faster SDXL.

Deepfaketown and Botpocalypse Soon. Wherever would we make deepfakes?

They Took Our Jobs. Hey, those jobs are only for our domestic robots.

Get Involved. Peter Wildeford is hiring. Send in your opportunities, folks!

Introducing. Sure, Graph of Thoughts, why not?

In Other AI News. AI gives paralyzed woman her voice back, Nvidia invests.

China. New blog about AI safety in China, which is perhaps a thing you say?

The Best Defense. How exactly would we defend against bad AI with good AI?

Portents of Gemini. It is coming in December. It is coming in December.

Quiet Speculations. A few other odds and ends.

The Quest for Sane Regulation. CEOs to meet with Schumer, EU's AI Act.

The Week in Audio. Christiano and Leahy give talks, Rohit makes his case.

Rhetorical Innovation. Some relatively promising attempts.

Llama No One Stopping This. Meta to open source all Llamas no matter what.

No One Would Be So Stupid As To. Bingo, sir.

Aligning a Smarter Than Human Intelligence is Difficult. Davidad has a plan.

People Are Worried About AI Killing Everyone. Roon, the better critic we need.

Other People Are Not As Worried About AI Killing Everyone. Consciousness?

The Wit and Wisdom of Sam Altman. Do you feel lucky? Well, do ya?

The Lighter Side. The big time.

Language Models Offer Mundane Utility

A class on the economics of ChatGPT, complete with podcast recording. More like this, please, no matter my quibbles. I especially don't think survey courses, in economics or elsewhere, are the way to go. Focus on what matters and do something meaningful rather than try to maximize gesturing. If you let me teach students with other majors one economics class, teach them the basics of micro and then use that to explore what matters sounds like a great plan. So is getting students good at using LLMs.

Use algorithmic instructions to let LLMs accurately do tasks like 19-digit addition.

Summarize writing. It seems GPT-4 summaries are potentially more accurate than humans ones.

We encountered two practical problems:

Not following instructions. Bigger models were better at following instructions. We had to use another LLM to understand the outputs of the smaller LLMs and work out if it said A or B was the answer.

Ordering bias. Given A and B, are you more likely to suggest A simply because it is first? One way to test this is to swap the ordering and see how many times you say A both times or B both times.

Once we dealt with these problem we saw:

Human: 84% (from past research)

gpt-3.5-turbo: 67.0% correct (seemed to h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nuclear safety/security: Why doesn't EA prioritize it more?, published by Rockwell on August 31, 2023 on The Effective Altruism Forum.

I'm dissatisfied with my explanation of why there is not more attention from EAs and EA funders on nuclear safety and security, especially relative to e.g. AI safety and biosecurity. This has come up a lot recently, especially after the release of Oppenheimer. I'm worried I'm not capturing the current state of affairs accurately and consequently not facilitating fully contextualized dialogue.

What is your best short explanation?

(To be clear, I know many EAs and EA funders are working on nuclear safety and security, so this is more so a question of resource allocation, rather than inclusion in the broader EA cause portfolio.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Michael Nielsen remarks on 'Oppenheimer', published by Tom Barnes on August 31, 2023 on The Effective Altruism Forum.

This is a linkpost to a recent blogpost from Michael Nielsen, who has previously written on EA among many other topics. This blogpost is adapted from a talk Nielsen gave to an audience working on AI before a screening of Oppenheimer. I think the full post is worth a read, but I've pulled out some quotes I find especially interesting (bolding my own)

I was at a party recently, and happened to meet a senior person at a well-known AI startup in the Bay Area. They volunteered that they thought "humanity had about a 50% chance of extinction" caused by artificial intelligence. I asked why they were working at an AI startup if they believed that to be true. They told me that while they thought it was true, "in the meantime I get to have a nice house and car".

[...] I often meet people who claim to sincerely believe (or at least seriously worry) that AI may cause significant damage to humanity. And yet they are also working on it, justifying it in ways that sometimes seem sincerely thought out, but which all-too-often seem self-serving or self-deceiving.

Part of what makes the Manhattan Project interesting is that we can chart the arcs of moral thinking of multiple participants [...] Here are four caricatures:

Klaus Fuchs and Ted Hall were two Manhattan Project physicists who took it upon themselves to commit espionage, communicating the secret of the bomb to the Soviet Union. It's difficult to know for sure, but both seem to have been deeply morally engaged and trying to do the right thing, willing to risk their lives; they also made, I strongly believe, a terrible error of judgment. I take it as a warning that caring and courage and imagination are not enough; they can, in fact, lead to very bad outcomes.

Robert Wilson, the physicist who recruited Richard Feynman to the project. Wilson had thought deeply about Nazi Germany, and the capabilities of German physics and industry, and made a principled commitment to the project on that basis. He half-heartedly considered leaving when Germany surrendered, but opted to continue until the bombings in Japan. He later regretted that choice; immediately after the Trinity Test he was disconsolate, telling an exuberant Feynman: "It's a terrible thing that we made".

Oppenheimer, who I believe was motivated in part by a genuine fear of the Nazis, but also in part by personal ambition and a desire for "success". It's interesting to ponder his statements after the War: while he seems to have genuinely felt a strong need to work on the bomb in the face of the Nazi threat, his comments about continuing to work up to the bombing of Hiroshima and Nagasaki contain many strained self-exculpatory statements about how you have to work on it as a scientist, that the technical problem is too sweet. It smells, to me, of someone looking for self-justification.

Joseph Rotblat, the one physicist who actually left the project after it became clear the Nazis were not going to make an atomic bomb. He was threatened by the head of Los Alamos security, and falsely accused of having met with Soviet agents. In leaving he was turning his back on his most important professional peers at a crucial time in his career. Doing so must have required tremendous courage and moral imagination. Part of what makes the choice intriguing is that he himself didn't think it would make any difference to the success of the project. I know I personally find it tempting to think about such choices in abstract systems terms: "I, individually, can't change systems outcomes by refusing to participate ['it's inevitable!'], therefore it's okay to participate". And yet while that view seems reasonable, Rotblat's example shows it is incorrect.

His private moral...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should I patent my cultivated meat method?, published by Michelle Hauser on August 31, 2023 on The Effective Altruism Forum.

I am a third-year PhD candidate working on cultivated meat. In my research, I have developed a new and original approach to making cultivated meat that has the potential of being highly scalable and help reach economic parity, more than most other known methods that companies are working on.

My supervisor has left it up to me to decide whether to patent this method or not. I have half-based beliefs that patenting would be counter-productive in helping cultivated meat reach the market sooner, but I acknowledge that I am not knowledgeable enough in IP law/structures to really know.

If I do decide to patent the method, the patent will mostly belong to and be written by the Tech Transfer office at my university, which is of course for profit. I am not sure I trust them to consider my wish to make a fairly non-limiting patent. The usual course of this kind of thing is then for the tech transfer office to open a private startup to further develop the method. In any case, I intend to publish the method in a scientific journal. If we patent it, then it would be published after filing.

What would be best for the field as a whole - a scenario where scientists patent their findings and then publish them, or where they just publish everything open source? Given that most scientists do patent and keep everything secret within companies, what would be best to do in my position?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Progress report on CEA's search for a new CEO, published by MaxDalton on August 31, 2023 on The Effective Altruism Forum.

I wanted to give an update on the Centre for Effective Altruism (CEA)'s search for a new CEO.

We (Claire Zabel, Max Dalton, and Michelle Hutchinson) were appointed by the Effective Ventures boards to lead this search and make a recommendation to the boards. The committee is advised by James Snowden, Caitlin Elizondo, and one experienced executive working outside EA.

We previously announced the search and asked for community input in this post.

Note, we set out searching for an Executive Director, and during the process have changed the role title to CEO because it was more legible to candidates not familiar with CEA or EV. The role scope remains unchanged.

In summary, we received over 400 nominations, reached out to over 150 people, spoke to about 60, and received over 25 applications. We're still considering around 15 candidates, and are currently more deeply assessing <5 candidates who seem especially promising.

Process

Over 400 people were nominated for the role

We invited recommendations from CEA staff, stakeholders, and the EA community via the Forum.

We selected candidates for outreach according to their profile and experience, the strength of recommendation we received, and (in some cases) our pre-existing knowledge about them.

We decided this approach (rather than reaching out to all nominees or having an open call for applications) because

Many nominations were fairly speculative (for instance one person nominated over 30 people, many of whom did not have management experience: this was helpful because it generated some useful names we would otherwise not have known about, but it probably would not have been a good use of our/candidates' time to reach out to all of these people).

More generally, we wanted to be careful about our time and applicants' time.

We thought (based on advice from professional recruiters) that a narrower outreach process would be more likely to be attractive to our most promising candidates (who matter disproportionately).

Overall our sense from talking to people with experience of this is that this selective, non-public process is fairly standard practice in executive recruitment. I think that the main reason for this is that in executive recruitment your top candidates (who are the most important ones) have a very high opportunity cost for their time, and personalized non-public outreach is a much stronger signal that it will be worth their time engaging in the process.

My guess is that some of our top candidates would not have proactively submitted an application, but did engage when we reached out to them.

We had a lower bar for reaching out to candidates we were less familiar with (because a small chance that they would be our top candidate justified the effort).

In the end, we reached out to over 150 potential candidates

We shared a document containing information on CEA and the role, and invited them to meet a member of the search committee (typically Max).

We sometimes asked people to consider booking a call even if they weren't interested in the role, because we wanted to get their advice on our hiring process.

In many cases, we tried to personalize outreach messages somewhat and use our network:

When we knew the candidates, this was easy

Sometimes we asked for introductions from mutual connections

However there were some cases where we didn't personalize (e.g. if we couldn't find a good mutual connection).

To give a sense of the range of candidates we reached out to, according to our very rough assessment they had:

EA context

~50% high (e.g. they've been working at an EA org or otherwise have lots of signs of engagement with EA ideas).

~40% medium (e.g. they attended an EAG once or did 80k coaching, but n...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New Jury Analysis of the Smithfield Piglet Rescue Trial, published by JLRiedi on August 31, 2023 on The Effective Altruism Forum.

Faunalytics analyzed transcripts from interviews with jurors of the Smithfield Foods criminal trial - in which two animal rights activists were found not guilty of "stealing" two piglets from a factory farm in Utah. This qualitative analysis will help advocates understand why jurors sided with the defense, how to potentially apply these findings to future trials, and what forms of animal activism are most convincing.

Key Findings:

The "not guilty" verdict hinged, in part, on the monetary value of the piglets to Smithfield, which was argued to be less than zero. The piglets required veterinary care that exceeded their value to Smithfield. The jury was initially hesitant to say the piglets had no worth because they saw them as having inherent worth as living beings, however they ultimately decided the theft charges hinged on monetary value only.

The jury members believed the defendants, Wayne and Paul, did not have the intent to steal. Before their investigation of the Smithfield facility, Wayne said on video "if there's something we'll take it." The jury interpreted the "if" as meaning the two activists did not enter the facility knowing they'd have the opportunity to take piglets. However, one juror noted that if the defendants had a pattern of doing this in the past, the jury might have been more likely to find them guilty.

The participants all reported being more receptive to animal advocacy and animal welfare after the trial. One participant reported that they no longer eat ham. Another reported that while they still believe that pigs are here to be eaten, as a result of the trial they now believe that pig welfare should be improved. Another was even inspired to pursue animal activism.

Despite what media coverage indicates, the "right to rescue" was not a major factor in the jury's decision. Some media outlets (such as The Intercept and Democracy Now!) have characterized this trial as a test case for the "right to rescue" argument - the idea that one should be able to rescue animals, sometimes farmed animals, from distressing conditions. However, only two jurors mentioned this concept at all, and no jurors mentioned this idea as critical.

Background

The Smithfield Trial refers to the prosecution of two animal advocates who were charged with felony theft and burglary after they removed two piglets from a Smithfield Foods facility in Utah, United States.

Wayne Hsiung and Paul Darwin Picklesimer, a co-founder and member of Direct Action Everywhere, respectively, are activists "working to achieve revolutionary social and political change for animals in one generation." In 2017, Wayne and Paul entered the Circle Four Farms facility in Milford, Utah, and removed two injured piglets (later named Lily and Lizzie). Circle Four Farms is one of the largest industrial pig processing facilities in the United States and a subsidiary of Smithfield Foods, which is the world's largest pork producer. Once rescued, the piglets were provided with veterinary care and relocated to a sanctuary where they currently reside. The removal of the piglets was filmed and posted on social media under the title "Operation Deathstar."

In September 2022, Wayne and Paul went on trial in Washington County, Utah on charges of felony theft and burglary for removing the piglets. They were acquitted (i.e., found not guilty) by a jury on both counts.

This trial may interest animal advocates because it provides potential guidance for future trials and investigations. Additionally, this analysis provides insight as to which pro-animal arguments are more persuasive more generally.

In this study, we analyzed themes from interviews with five Smithfield Trial jury members (referred t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long-Term Future Fund Ask Us Anything (September 2023), published by Linch on August 31, 2023 on The Effective Altruism Forum.

LTFF is running an Ask Us Anything! Most of the grantmakers at LTFF have agreed to set aside some time to answer questions on the Forum.

I (Linch) will make a soft commitment to answer one round of questions this coming Monday (September 4th) and another round the Friday after (September 8th).

We think that right now could be an unusually good time to donate. If you agree, you can donate to us here.

About the Fund

The Long-Term Future Fund aims to positively influence the long-term trajectory of civilization by making grants that address global catastrophic risks, especially potential risks from advanced artificial intelligence and pandemics. In addition, we seek to promote, implement, and advocate for longtermist ideas and to otherwise increase the likelihood that future generations will flourish.

In 2022, we dispersed ~250 grants worth ~10 million. You can see our public grants database here.

Related posts

LTFF and EAIF are unusually funding-constrained right now

EA Funds organizational update: Open Philanthropy matching and distancing

Long-Term Future Fund: April 2023 grant recommendations

What Does a Marginal Grant at LTFF Look Like?

Asya Bergal's Reflections on my time on the Long-Term Future Fund

Linch Zhang's Select examples of adverse selection in longtermist grantmaking

About the Team

Asya Bergal: Asya is the current chair of the Long-Term Future Fund. She also works as a Program Associate at Open Philanthropy. Previously, she worked as a researcher at AI Impacts and as a trader and software engineer for a crypto hedgefund. She's also written for the AI alignment newsletter and been a research fellow at the Centre for the Governance of AI at the Future of Humanity Institute (FHI). She has a BA in Computer Science and Engineering from MIT.

Caleb Parikh: Caleb is the project lead of EA Funds. Caleb has previously worked on global priorities research as a research assistant at GPI, EA community building (as a contractor to the community health team at CEA), and global health policy.

Linchuan Zhang: Linchuan (Linch) Zhang is a Senior Researcher at Rethink Priorities working on existential security research. Before joining RP, he worked on time-sensitive forecasting projects around COVID-19. Previously, he programmed for Impossible Foods and Google and has led several EA local groups.

Oliver Habryka: Oliver runs Lightcone Infrastructure, whose main product is Lesswrong. Lesswrong has significantly influenced conversations around rationality and AGI risk, and the LWits community is often credited with having realized the importance of topics such as AGI (and AGI risk), COVID-19, existential risk and crypto much earlier than other comparable communities.

You can find a list of our fund managers in our request for funding here.

Ask Us Anything

We're happy to answer any questions - marginal uses of money, how we approach grants, questions/critiques/concerns you have in general, what reservations you have as a potential donor or applicant, etc.

There's no real deadline for questions, but let's say we have a soft commitment to focus on questions asked on or before September 8th.

Because we're unusually funding-constrained right now, I'm going to shill again for donating to us.

If you have projects relevant to mitigating global catastrophic risks, you can also apply for funding here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Responses to apparent rationalist confusions about game / decision theory, published by Anthony DiGiovanni on August 31, 2023 on LessWrong.

I've encountered various claims about how AIs would approach game theory and decision theory that seem pretty importantly mistaken. Some of these confusions probably aren't that big a deal on their own, and I'm definitely not the first to point out several of these, even publicly. But collectively I think these add up to a common worldview that underestimates the value of technical work to reduce risks of AGI conflict. I expect that smart agents will likely avoid catastrophic conflict overall - it's just that the specific arguments for expecting this that I'm responding to here aren't compelling (and seem overconfident).

For each section, I include in the footnotes some examples of the claims I'm pushing back on (or note whether I've primarily seen these claims in personal communication). This is not to call out those particular authors; in each case, they're saying something that seems to be a relatively common meme in this community.

Summary:

The fact that conflict is costly for all the agents involved in the conflict, ex post, doesn't itself imply AGIs won't end up in conflict. Under their uncertainty about each other, agents with sufficiently extreme preferences or priors might find the risk of conflict worth it ex ante. (more)

Solutions to collective action problems, where agents agree on a Pareto-optimal outcome they'd take if they coordinated to do so, don't necessarily solve bargaining problems, where agents may insist on different Pareto-optimal outcomes. (more)

We don't have strong reasons to expect AGIs to converge on sufficiently similar decision procedures for bargaining, such that they coordinate on fair demands despite committing under uncertainty. Existing proposals for mitigating conflict given incompatible demands, while promising, face some problems with incentives and commitment credibility. (more)

The commitment races problem is not just about AIs making commitments that fail to account for basic contingencies. Updatelessness (or conditional commitments generally) seems to solve the latter, but it doesn't remove agents' incentives to limit how much their decisions depend on each other's decisions (leading to incompatible demands). (more)

AIs don't need to follow acausal decision theories in order to (causally) cooperate via conditioning on each other's source code. (more)

Most supposed examples of Newcomblike problems in everyday life don't seem to actually be Newcomblike, once we account for "screening off" by certain information, per the Tickle Defense. (more)

The fact that following acausal decision theories maximizes expected utility with respect to conditional probabilities, or counterfactuals with the possibility of logical causation, doesn't imply that agents with acausal decision theories are selected for (e.g., acquire more material resources). (more)

Ex post optimal =/= ex ante optimal

An "ex post optimal" strategy is one that in fact makes an agent better off than the alternatives, while an "ex ante optimal" strategy is optimal with respect to the agent's uncertainty at the time they choose that strategy. The idea that very smart AGIs could get into conflicts seems intuitively implausible because conflict is, by definition, ex post Pareto-suboptimal. (See the "inefficiency puzzle of war.")

But it doesn't follow that the best strategies available to AGIs given their uncertainty about each other will always be ex post Pareto-optimal. This may sound obvious, but my experience with seeing people's reactions to the problem of AGI conflict suggests that many of them haven't accounted for this important distinction.

As this post discusses in more detail, there are two fundamental sources of uncertainty (o...