Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monitoring for deceptive alignment, published by Evan Hubinger on September 8, 2022 on The AI Alignment Forum.

This post is a follow-up to “AI coordination needs clear wins.” Thanks to Ethan Perez, Kate Woolverton, Richard Ngo, Anne le Roux, Sam McCandlish, Adam Jermyn, and Danny Hernandez for useful conversations, comments, and feedback.

In this post, I want to propose a clear, concrete coordination task that I think might be achievable soon given the current landscape, would generate a clear coordination win, and that I think would be highly useful in and of itself. Specifically:

I want DeepMind, OpenAI, and Anthropic to commit to actively monitor and look for evidence of deceptive alignment in their models—as well as run experiments to try to predict when and where deceptive alignment might occur before it does.

Notably, I am specifically referring only to the narrow case of deceptive alignment here, not just any situation where models say false things. Deceptive alignment is specifically a situation where the reason the model looks aligned is because it is actively trying to game the training signal for the purpose of achieving some ulterior goal.

I think this is a pretty minimal ask that would nevertheless be a clear win. Among all AI failure modes, deceptive alignment is one of the most unambiguously bad, which means, though I expect lots of disagreement on its likelihood, there should be little disagreement regarding its severity. Furthermore, things that I’m not asking for:

that they make such a commitment highly public or legally binding,

that they commit to any specific way of monitoring for or addressing the problem,

that any organization has to share anything they find with any other organization, or

that they commit to anything other than testing and monitoring.

Nevertheless, as I stated, I think this would still be quite useful.

First, I think deceptive alignment is clearly one of the most concerning problems in AI alignment. Though they don’t always use the framing of deceptive alignment specifically, most presentations of AI risk feature the phenomenon of an AI that wants something in the world deceiving our oversight procedures to get that something. See: treacherous turns, schemers, influence-seeking behavior, power-seeking AI. Furthermore, I think there is a strong case for why deceptive alignment is reasonably likely to appear by default in advanced machine learning systems.

Second, I think that it is plausible that, if deceptive alignment is eventually going to be a problem, we’ll be able to see it and produce demonstrations of it before it becomes existentially dangerous. I’m less confident in this premise than the previous one, but my current sense is that the space of deceptively aligned models is large enough to include lots of deceptively aligned models that would be willing to display their deception in the presence of the right sort of honeypot. For example, you might be able to find deceptively aligned models that:[1]

have relatively short time horizons such that they are very willing to defect even if their payoff and/or probability of successful defection is low,

are confused about what level of simulation they’re in such that they defect to break out of an inner simulation only to find themselves in an outer simulation,

are bad enough at hiding their deception that simple transparency tools can detect them,

are unwilling to cooperate with other deceptively aligned models to all defect simultaneously,

etc.

Of course, just being able to detect deception doesn’t mean we’ll be able to solve it—and in fact in many cases it seems likely that our default ways of addressing deception would result in deception that just hides itself better—but I still think that being able to detect it before it becomes catastrophic would be a huge win, since i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [An email with a bunch of links I sent an experienced ML researcher interested in learning about Alignment / x-safety.], published by David Scott Krueger on September 8, 2022 on The AI Alignment Forum.

[[FYI: I'm just copying this in and removing a few bits; apologies for formatting; I don't intend to post the attachments]]EtA: I (embarrasingly and unfortunately) understated Richard Ngo's technical background in email; I left what I originally wrote in strikethrough.EtA(2): I thought a bit more context would be useful here:- This email was designed for a particular person after several in person conversations.- I'm not proposing this as anything like "The best things to send to an arbitrary 'experienced ML researcher interested in learning about Alignment / x-safety'".- I didn't put a ton of effort into this.- I aimed to present a somewhat diverse and representative sampling of x-safety stuff.EtA (3): Suggestions / Feedback welcome!OK I figure perfect is the enemy of the good and this is already a doozy of an email, so I'm just going to send it :)I think it would be great to get a better sense of what sort of materials are a good introduction for someone in your situation, so please

let me know what you find most/least useful/etc.!A few top recommendations...

[[FYI: I'm just copying this in and removing a few bits; apologies for formatting; I don't intend to post the attachments]]EtA: I (embarrasingly and unfortunately) understated Richard Ngo's technical background in email; I left what I originally wrote in strikethrough.EtA(2): I thought a bit more context would be useful here:- This email was designed for a particular person after several in person conversations.- I'm not proposing this as anything like "The best things to send to an arbitrary 'experienced ML researcher interested in learning about Alignment / x-safety'".- I didn't put a ton of effort into this.- I aimed to present a somewhat diverse and representative sampling of x-safety stuff.EtA (3): Suggestions / Feedback welcome!OK I figure perfect is the enemy of the good and this is already a doozy of an email, so I'm just going to send it :)I think it would be great to get a better sense of what sort of materials are a good introduction for someone in your situation, so please let me know what you find most/least useful/etc.!A few top recommendations...

- read about DeepMind's "safety, robustness, and assurance" breakdown of alignment- sign up for the Alignment and ML Safety newsletters, and skim through the archives

- Look at this syllabus I mocked up for my UofT application (attached). I tried to focus on ML and include a lot of the most important ML papers, although there's a lot missing.

- Read about RL from human preferences if you haven't already. I imagine you might've seen the "backflipping noodle" blog post. I helped author a research agenda based on such approaches Scalable agent alignment via reward modeling. The research I talked about in my talk is part of a project on understanding reward model hacking; we wrote a short grant proposal for that (attached). I'm in the midst of rethinking how large a role I expect reward modeling (or RL more generally) to play in future AI systems, but I previously considered this one of the highest priority directions to work on. Others (e.g. Jan, first author of the agenda) are working on showing what you can do with reward modeling; I'm more concerned with figuring out if it is a promising approach at all or not (I suspect not, because of power-seeking / instrumental convergence).

- Look at the "7 Alternatives for agent alignment" in the agenda for a brief overview of alternative approaches to specification.- Look at "10.1 Related research agendas" in ARCHES for a quick overview of various research agendas. - The most widely known/cited agenda is

Concrete P...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Maximum Impact Fund is now the Top Charities Fund, published by GiveWell on September 8, 2022 on The Effective Altruism Forum.

Author: Elie Hassenfeld, GiveWell Co-Founder and Chief Executive Officer

We’ve decided to rename the Maximum Impact Fund to better describe what opportunities this fund supports. The Maximum Impact Fund will now be called the Top Charities Fund.

We recently announced changes to our top charity criteria that include a new requirement for our top charities: that we have a high degree of confidence in our expectations about the impact of their programs. Alongside this update, we also introduced a new giving option, the All Grants Fund. The All Grants Fund supports the full range of GiveWell’s grantmaking and can be allocated to any grant that meets our cost-effectiveness bar—including opportunities outside of our top charities and riskier grants with high expected value.[1]

The new All Grants Fund is a complement to what we have called our Maximum Impact Fund, which is granted to cost-effective opportunities among our top charities. However, we’ve received feedback that describing the fund that supports grantmaking only to our top charities as having “Maximum Impact” is confusing in light of the opportunity to support a wider range of opportunities (with potentially higher expected value) through the All Grants Fund.

Based on this feedback, we’ve decided to change the name of the Maximum Impact Fund to the Top Charities Fund.

Only the name is changing—we aren’t making any further changes to the underlying fund beyond those discussed in our previous post, so any incoming donations designated for the Maximum Impact Fund will automatically be allocated to the Top Charities Fund. We will continue to use donations to this fund to support the highest-priority funding needs among our top charities each quarter. As before, we will apply the same cost-effectiveness bar across our grantmaking, regardless of whether the funding comes from the All Grants Fund or Top Charities Fund.

We appreciate everyone who provided feedback and hope that the new name provides more clarity about how donations to our funds will be used.

More details about our funds and their impact are here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 9/8/22: Booster Boosting, published by Zvi on September 8, 2022 on LessWrong.

Does it make sense to get the new updated booster? I give my (Not Medical Advice!) definitive take on how I would personally make the decision, and report the opinions of a number of other sources. Unless something changes, this is my take on that subject.

There’s also a variety of small things happening, both in the Covid section and later in the non-Covid section. I get to do two distinct little notes on gambling, and the FDA Delenda Est of the week is about the Food rather than the Drugs.

Executive Summary

Updated boosters available, consider getting one.

Still a lot of rather silly mask mandates, although no longer in NYC.

A rather impressive (if not impactful) case of pure Chutzpah.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week (HOLIDAY): 420k deaths (-20%) and 2,550 deaths (-20%).

Results: 452k cases (-14%) and 2,728 deaths (-17%).

Prediction for next week: 490k cases (+9%) and 3,000 deaths (+10%).

This prediction is expecting some real improvement, but for it to be dominated by a larger reversion from a holiday weekend.

Deaths

Cases

Booster Boosting

Public service announcement before the main discussion:

With that out of the way: To be as explicit as possible, while keeping in mind this is Not Medical Advice, I will give my full answer to how I personally for myself would answer ‘should I get the new updated booster when it becomes available?’

Essentially, I would get the booster if and only if I was already vaccinated, had not been infected or vaccinated/boosted within 6 months, and either (1) not yet had a first booster or (2) was at relatively high risk.

So essentially:

If I had not yet been vaccinated (first 2 shots), I’d get vaccinated first.

If I had been infected or vaccinated in the last 6 months, I’d wait.

If I had been vaccinated at least 6 months ago, but not boosted, and not infected in the last 6 months to my knowledge, then yes, I’d get boosted.

If I had already been vaccinated and boosted more than six months ago, and had not to my knowledge been infected in the past 6 months, and was in poor health or are over the age of 50 or so, yes I would get a second booster.

If I had already been vaccinated and boosted more than six months ago, and had not to my knowledge been infected in the past 6 months, and was not in poor health or are over the age of 50 or so, probably no, but it would be a small mistake either way.

I would strictly prefer to get the new booster rather than the old booster.

I would consider it reasonable to attempt to ‘time’ the booster to match plans to visit vulnerable others and/or wait until a larger wave was coming. Note that if a large wave does come there is a good chance it will be a new strain.

White House says Americans will ‘only’ need yearly booster going forward. NPR offers a variety of takes on where the thresholds should be and whether to time things for a bigger wave. Celine Gounder suggests focusing on timing your booster to be 2-4 weeks before visiting vulnerable relatives if you are healthy, and splitting up the Covid and Flu shots if you can. Your Local Epidemiologist couldn’t be more excited to get hers, and has detailed thoughts on exactly how long to wait after various things.

My decision means I will officially not be ‘up to date’ on my booster. Neither will be (for a time) Dr. Fauci, despite his willingness to get a third booster.

The question that matters here: What does it mean to be ‘up to date’?

If ‘up to date’ purely meant ‘has had all the boosters, all that one could possibly have’ as a supererogatory aspiration the way someone might say they are ‘up to date’ on the entire Marvel Cinematic Universe, then this definition is fine. You have, for now, watched all the movies and TV shows or had all...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Save the Date: EAGx LatAm, published by LGlez on September 8, 2022 on The Effective Altruism Forum.

EAGx LatAm will take place in Mexico City at the Casa de la Bola Museum, 6th-8th of January. One of our goals is to connect the LatAm community with the broader international community. So, although this event is primarily for the Spanish- and Portuguese-speaking EA communities, applications from experienced members of the international community are very welcome! What better way to start 2023?

This will overlap with the EA Fellowships and Visitors Programme that we’re running in Mexico 1st Nov- 30th Jan with the aim of kickstarting an EA Hub in the city. So expect great EA vibes on top of fantastic local (vegan) cuisine and fun cultural activities like city tours and regional dance lessons.

Applications will open in October.

Why EAGx LatAm?

1) The EA movement in LatAm has grown over the last couple of years, with new local and university groups in Mexico, Chile and Colombia joining already well-established groups like EA Brazil. This EAGx is an occasion to foster a sense of community and connect people who are facing similar opportunities and challenges.

2) Many people from Latin America experience barriers to attending international EAG(x) conferences in the US or Europe, including visa problems and inconvenient and expensive international connections. As a result, the international community is missing out on talent and insights from a sizable part of the population, while the LatAm community misses on opportunities and valuable networks.

With this in mind, the conference aims to:

-Raise the profile of EA in LatAm.

-Create and strengthen connections between EAs within LatAm, the Spanish- and Portuguese-speaking communities.

-Create and strengthen connections between EAs in LatAm and the rest of the world.

Who is this conference for?

This conference is for you if you.:

-Are from LatAm, new to EA and looking forward to learning more and connecting with like-minded people. Most of our talks will be introductory and cover a broad range of topics.

-Are from LatAm, very much not new to EA, and excited to meet other members of the community.

-Are an experienced member of the community from outside LatAm and keen to connect with EAs who might be under your radar.

I don’t speak Spanish, is this event for me?

Yes! First of all, although the lead organisers (Sandra Malagón and Laura González) are also the coordinators of the Spanish-speaking community, we’re in touch with the Brazilian community, and our goal is to welcome people interested in EA across LatAm and the world at large, regardless of language(s) spoken. Most talks will use English as a lingua franca, with the exception of a few topic-specific talks and workshops like “Community building in the Portuguese/Spanish-speaking world”, which will be run in the relevant languages. We will have a system of stickers or colour-coded tags for participants to signal the languages they speak.

If you’re unsure about whether to apply, err on the side of applying.

If you have any questions or comments don’t hesitate to write to [email protected] . We accept emails in English, Spanish and Portuguese :)

See you in Mexico!

The Organising Team

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Suggestions for how to talk about pregnancy losses, published by Michelle Hutchinson on September 8, 2022 on The Effective Altruism Forum.

The advice on how to talk to a person who’s experienced a pregnancy loss seems generally pretty bad to me. That’s a great shame because these losses are a hard thing to talk about but also a common occurrence. Having had a miscarriage and a late term stillbirth, I’ve thought a bit about ways of communicating about them in ways that land well for people, and I thought that might be useful to share.

‘Pregnancy loss’ covers a wide range of experiences, including early stage miscarriages, abortions and still births. Most of this is aimed at how to talk to the person who was physically pregnant.

What follows is most applicable for people you know well - for talking to friends or colleagues. If you’re meeting someone you don’t know too well at an event who you know has suffered a pregnancy loss, it’s quite likely that the right approach is simply to let them go about the event as normal, rather than bringing it up. Talking about it can be pretty hard for people, particularly with someone they don’t know that well and when they weren’t expecting to.

People’s experiences differ widely

People experience pregnancy losses really differently on the spectrum from ‘I’m sick in a particularly painful/private way and my future suddenly looks really different’ to ‘a person I’ve been thinking a lot about for months has died’. That means people’s ideal way of talking about it / being comforted is pretty different. On the other end you want to focus just on the people losing the pregnancy: ‘How are you feeling? Are you in a lot of pain?’. On the other end are things like ‘I’m sorry for your loss. Did you name her?’.

The ideal is to figure out how the person is experiencing the pregnancy loss, and try to mirror that. Before that, you might want to keep with general comments along the lines of ‘I’m really sorry; that must be really tough; how are you doing?’.

Also, people’s experiences can change day to day. And they don’t necessarily even know themselves how they’re experiencing it. So the ideal is to be open-mindedly sympathetic and hold space for the person to feel anywhere from fine to awful.

Beware of escalating grief narratives

A failure mode that happened with me a bunch after the stillbirth was a kind of escalating cycle of everyone upping the amount to treat the pregnancy loss as a death. For people talking to me, it may have seemed better to err on the side of acknowledging the loss of a baby because it’s worse to fail to acknowledge a loss than to acknowledge one that didn’t happen. But that caused me to feel callous to not match their level of acknowledging a loss. That’s actually pretty bad. The reason is that the people undergoing the pregnancy loss don’t actually know how to feel about it, feel differently at different times, and so can be quite affected by different narratives. The more other people treat it like someone you know died, the more it can feel like they did.

That can feel more sad than the alternative (because you start feeling an increased sense of loss) or it can cause guilt (‘given that someone I know died, shouldn’t I be feeling more grief for them and less upset about the effects on me and my future?’).

As far as I can tell it used to be that people’s sense of ‘losing a person’ was mostly ignored, so now there are various things put in place to try to compensate for that, and medical staff are trained to try to acknowledge people might be feeling it as a death. Society also is more used to talking about good things and also to pregnancies being successful. So narratives that avoid talking about the pregnancy loss as a death feel less available/salient.

Specific narratives people might be experiencing

Here’s some ways p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solar Blackout Resistance, published by jefftk on September 8, 2022 on LessWrong.

Many houses have solar panels, but in a power outage almost all of them shut down. Especially with serious blackouts like Texas had or like Europe may have this winter, getting even minimal power during an outage is valuable. Designing residential grid-tied rooftop solar so that it only worked when the grid is up was a serious mistake that squandered what could have been a major benefit of putting panels on roofs. Let's fix this!

When the grid goes down it's important that solar panels don't keep sending out power, which could shock utility workers fixing things. Having inverters shut themselves off is the easiest way to ensure that never happens, but it's not the only way: the inverter could disconnect from the grid while providing power just to the house. This would only work when the sun was shining, of course, and only when there was enough sunlight on the panels for what you were trying to run. A large system in a sunny area could be set up to try to power the whole house, while a less productive system might be set up to power a few priority circuits or even just a single outlet.

Making systems work this way by default would have major advantages in a range of disasters. You would immediately have widespread distributed generation, and so many things rely on electricity:

Communications. Really important for people to coordinate. Even just a small amount of power for recharging phones would go a long way.

Medical devices. Many people rely on

ventilators, powered wheelchairs, etc, and these generally only have a few hours of battery if the grid goes down.

Refrigeration. Keeping food (and medicine) from spoiling.

Heating. Even gas and oil heat need electricity for blowers, pumps, etc. and generally will not function at all when the power is out.

Cooling. Fans, AC, etc. Combines especially well with solar, since cooling need is highest when available solar is highest.

Pumps, tools, etc. Even more important during disasters.

Looking ahead to likely winter power outages in Europe, and potential outages here if things get worse [1] it would be far better if these millions of rooftop solar installs were able to operate independently.

You can install systems with batteries, which would be more robust than relying on however much power the sun happens to be producing at the moment, but large batteries are expensive and I'm not aware of anyone selling systems like this with small batteries. What if we used no battery?

Electrically, this is pretty practical. When we got solar I thought about this a lot and we installed an SMA

Sunny Boy with a "Secure Power Supply" (SPS) in the basement. It didn't end up costing much more: something like $250.

Unfortunately these aren't allowed anymore for residential rooftop solar. Starting with the 2014 National Electrical Code (NEC) you need to have some form of rapid shutdown to protect firefighters in an emergency, and in the 2017 revision it got much stricter to where the SPS functionality would have to be completely redesigned to qualify. The only option now, as far as I can tell, is the recently released Enphase

IQ8 Sunlight Backup. It's very expensive: I got some quotes and it added about $7k (23%).

I think it's likely that this rule change didn't consider the resiliency downsides of prohibiting the current versions of these systems, and it would have been better to make the change over a longer period with more notice to manufacturers. For example, it took until December 2021 for the first NEC 2017 compliant backup system, the IQ-8 to come out, 5 years from code publication. One way to get most of the benefits of the code change without this particular harm would have been to offer a temporary exception for island-capable solar.

My understanding is t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Developing Next Gen PPE to Reduce Biorisks, published by AndyGraham on September 8, 2022 on The Effective Altruism Forum.

TL;DR We are intending to run a feasibility study to establish requirements and a theory of change for novel ‘Super PPE’. Assuming this concludes positively we will develop Super PPE to reduce GCBR risk.

This post builds on the 11th Jan post on concrete biosecurity projects by ASB, ECA, and the ‘list of lists’ by TA, specifically with regard to ‘Super PPE’. Thank you to the authors of these posts and all the others that have helped with our research so far, particularly Vivian Belenky and their work with the NextGen PPE group, Cass Springer and his knowledge of the area, and Caleb Parikh for advice and for reviewing this post.

What is the context?

Global Catastrophic Biological Risks (GCBRs) present a known and well documented threat - both in an existential form, and as s-risks from lesser but still greatly damaging pandemics or other biological events [1].

Many data, infrastructure, logistical, and biomedically based methods are in development to reduce catastrophic biorisk [2] [3] [4] [5]. However, there is a concerning lack of development of novel and next generation hardware and physical equipment in the same space.

One particular item in this area is Biohazard Personal Protective Equipment. In any potentially catastrophic biological event that breaches initial containment efforts, we have extremely high confidence that widespread use of high protection level PPE, likely akin to positive pressure BSL4 suits, would be required for recovery and continuation of civilisation (we don’t have a source for this but literature in the area discusses it as if given).

Unfortunately, existing PPE of an appropriate protection level is not practical for widespread use during a catastrophic event. A quick list of some major issues with suits such as BSL4 suits and L5 Tychem suits:

Incredibly bulky and usually poorly fitting - Restrictive in motion and visibility

Incredibly hot & humid [6]. (This is often reported as an issue in climate controlled labs, let alone out in the real world; Ebola responders often couldn’t work in an L5 suit for more than one hour due to heat)

Prone to Damage of suit fabric (average of 3.5 damages over 6 months!) and hoses

Very Expensive: $1500-5000+

Most require external hoses and immobile plant

Difficult to put on and take off

Limited reusability (in many cases)

The development of ‘Super PPE’ was flagged earlier this year in the EA forums as a concrete biosecurity project. This is echoed by other organisations, such as the Apollo program for biodefence, which has identified next-generation PPE as a technology priority for the next decade.

This is a sorely overdue development that could see a huge leap in technology/equipment quality, simply due to the fact that designs haven’t significantly changed since 1979.

We are aware of some efforts that have been made to improve on full body biohazard PPE, particularly relating to and following the 2014-16 Ebola outbreak. Whilst these improvements are a brilliant start, there is still a long way to go until we reach global-catastrophe-ready suits.

What do we propose?

In order to make substantial progress on Super PPE, the authors are preparing to apply for funding to perform an initial feasibility study into the design and development of next generation PPE for biorisks. Should the study return favourably (we have high confidence that it will), it will be followed by further stages to prove the concept, create prototypes, and hopefully scale to a final product within 2-5 years. We are intending to do this primarily through an engineering consultancy that we run, which enables access to engineers across several engineering disciplines (including Mechanical, Chemical, Bio, and Materials) and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Should AI Owe To Us? Accountable and Aligned AI Systems via Contractualist AI Alignment, published by Xuan (Tan Zhi Xuan) on September 8, 2022 on The AI Alignment Forum.

This is an extended and edited transcript of the talk I recently gave at EAGxSingapore 2022. The title has been changed for easier searchability of "Contractualist AI Alignment".

Abstract: Artificial intelligence seems poised to alter our civilization in transformative ways. How can we align this development with our collective interests? Dominant trends in AI alignment research adopt a preference utilitarian conception of alignment, but this faces practical challenges when extended to a multiplicity of humans, values, and AI systems. This talk develops contractualist AI alignment as an alternative framework, charting out a vision of societal-scale alignment where AI systems can serve a plurality of roles and values, governed by and accountable to collectively decided, role-specific norms, with technical work ensuring compliance with these overlapping social contracts in the face of normative ambiguity.

Overview

This talk is an attempt to condense a lot of my thinking about AI alignment over the past few years, and why I think we need to orient the field towards a different set of questions and directions than have typically been pursued so far.

It builds upon many ideas in my previous talk on AI alignment and philosophical pluralism, as well as arguments in Comprehensive AI Services as General Intelligence, AI Research Considerations for Human Existential Safety (ARCHES), How AI Fails Us, and Gillian Hadfield's work on The Foundations of Cooperative Intelligence.

This will cover a lot of ground, so below is a quick overview:

The dominant "preference utilitarian" framework in AI alignment research.

Challenges for extending this framework to a multiplicity of humans, values, and autonomous systems.

Considerations and desiderata that a successful approach to society-scale AI alignment should address.

Pluralist and contractualist AI alignment as an alternate framework, including implications for governance, technical research, and philosophical foundations.

Alignment: A Preference Utilitarian Approach

One way of describing the framework that most alignment researchers implicitly adopt is a "preference utilitarian" approach. Stuart Russell’s 3 Principles for Beneficial AI are good summary of this approach. Recognizing that many dangers arise when machines optimize for proxy metrics that ultimately differ from human values, he instead advocates that:

The machine's only objective is to maximize the realization of human preferences.

The machine is initially uncertain about what those preferences are.

The ultimate source of information about human preferences is human behavior.

Stuart Russell, Human Compatible (2019)

More broadly, many researchers frame the problem as one of utility matching. Under certain assumptions, a single human’s preferences can be represented as a utility function over outcomes, and the goal is to build AI systems that optimize the same utility function.

(This is implicit in talk about, e.g., objective functions in inner misalignment, and reward modeling, which suggested that human objectives and values can ultimately be represented as a mapping to a scalar quantity called "reward" or "utility".)

Why is this hard? Because while it may be possible to ensure that the system does the right thing during development, it’s much harder to ensure this during deployment, especially as systems become more capable of achieving new outcomes. For example, a self-driving car might safely avoid obstacles for all situations it was trained on. But at deployment, the objective it’s effectively maximizing for might be much more positive than the true human objective, leading to car crashes.

So the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: First we shape our social graph; then it shapes us, published by Henrik Karlsson on September 7, 2022 on LessWrong.

The inside of a womb looks as it did 70,000 years ago, but the world outside has changed. In July 2021, when our daughter was born, the night sky didn’t light up with stars; it was lit up by the warm afterglow of sodium street lamps. Green-clad women carried the baby away, pumping oxygen into her mouth. It was like something out of a sci-fi: she had woken up, without a memory, in an alien world. Smeared in white-yellow fat, she didn’t know who she was nor what she was doing here. The only thing she knew, genetically, was that she needed to figure this out fast or die.

How do we ever do this?

Chimpanzees, who are born into the habitat their genes expect, get by largely on instinct. We cannot. We have to rely on what anthropologists call cultural learning. We have to observe the people that surround us; we have to figure out who among them navigate our local culture best and then extract the mental models that allow them to do so. This is a wicked problem. But we solve it instinctively. It is the main thing that sets us apart from chimpanzees.

As I wrote in Apprenticeship Online:

If you measure two-and-a-half-year-old children against [same-aged] chimpanzees and orangutans, they are about even in their capacity to handle tools and solve problems on their own. Only when it comes to observing others and repeating their actions is there a noticeable difference.

Two-and-a-half-year-olds can extract knowledge from people just by watching them move about a room. They start to desire what those around them desire. They pick up tacit knowledge. They change their dialect to match their peer groups. And after a handful of years of hanging about with people more skilled than themselves, our babies—these tiny, soft-skulled creatures—can out-compete chimpanzees in all but close combat.

This ability is not something you can turn on and off. You are always internalizing the culture around you. Even when you wish you didn’t. So you better surround yourself with something you want inside—curate a culture.

Your culture shapes who you become

How do you summon an interesting set of friends to bounce your ideas off? Where do you look to find good influences? Which types of output should you produce if you want people to route useful ideas your way?

This essay is the first in a series about culture curation. In later parts, I will go into detail about how to do this. But here I want to give a high-level view of why it makes sense to think about the world in this manner—as a graph you can restructure to change yourself. Why should you put effort into shaping your culture, rather than something else?

Here is a reason. Over the last few months, since May, I’ve helped Erik Hoel comb through the literature on the upbringings of historical geniuses. We’ve found a lot of interesting things, which Erik will cover in an upcoming post on his Substack. But what has struck me, more than anything else, is the insane quality of the cultures they internalized. The pedagogies their guardians employed differed radically; they had differing temperaments; they mastered different disciplines, but they all had this in common: they spent their days around highly competent people.

Most who grew up to become geniuses, pre-1900, were kept apart from same age peers and raised at home, by tutors or parents. Michel Montaigne’s father employed only servants who were fluent in Latin, curating a classical culture, so Montaigne would learn Latin as his mother tongue. J.S. Mill spent his childhood at his father’s desk, helping his father write a treatise on economics, running over to Jeremy Bentham’s house to borrow books and discuss ideas.

Blaise Pascal, too, was homeschooled by his father. His father choose...