Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI-kills-everyone scenarios require robotic infrastructure, but not necessary nanotech, published by avturchin on April 3, 2023 on LessWrong.

Tl;DR: Nanotech seems the only option for AI to kill everybody; but we can imagine a prosaic-AI-killing-everyone using three steps: taking over a country, building robotic infrastructure, releasing biological weapons.

Arguments why nanotech is needed for AI-kills-everyone

The concept of AI as an existential threat, as portrayed by EY, largely hinges on the development of manufacturing nanotechnology, which entails the creation of microscopic self-replicating robots capable of manipulating matter with atomic precision. This enables the "copying of a berry." A superintelligent AI must first establish a nanotech infrastructure to conquer the world and eliminate all humans, potentially by ordering protein-coding DNA sequences if it can calculate the correct sequences.

Criteria of robotic infrastructure

A nanotech infrastructure is essential for a comprehensive AI annihilation due to the following factors:

a) Concealment: AI's actions will remain undetected until the final strike.

b) Rapid elimination: AI will eradicate all humans instantaneously, preventing any interference.

c) Human-independence: Can operate without human involvement.

d) Swift construction: Nanotech can be developed rapidly, taking only a few weeks to synthesize required DNA strains and a few days for the initial nanobot replication. This grants a significant strategic advantage to the first AGI.

e) Route to superintelligence: Nanotech provides superior computational power compared to chips, enabling the first AI to rapidly ascend to an unparalleled superintelligence level. However, it is assumed that the first AI is already mildly superintelligent, as it can design the initial nanobots.

These factors also represent potential vulnerabilities that our last line of AI safety defense could target. For instance, shutting down DNA synthesis labs or developing our own nanotechnology and detection methods could reduce the efficacy of an AI-nanotech assault.

Few alternative AI infrastructure ideas possess all these characteristics, except perhaps:

a) A scenario where AI takeover occurs in a fully robotized world, with every household owning a home robot;

b) A form of biotechnology where AI can program biological organisms to execute tasks. However, this is a variation of nanotechnology, and AI computations cannot migrate into biological substrates.

Why these criteria?

The necessity for nanotech infrastructure in AI-kills-all situations arises from several factors:

If AI constructs a "conventional" robotic infrastructure, it will be visible and attacked before completion, increasing risks for the AI.

If AI cannot replace all humans, it remains vulnerable, as it requires a constant electricity supply, unlike humans. Destroying the electrical grid exposes AI to danger.

If AI cannot eradicate everyone instantaneously, humans will have time to retaliate.

If AI does not migrate to a nanotech-based computational substrate operating on independent energy sources, it remains dependent on a few data centers which are susceptible to airstrikes, sabotage, kill-switch codes, and power outages.

If AI does not gain computational and data advantages from nanotechnology, other AIs will soon achieve similar intelligence levels.

However, many people assign low prior probabilities to both nanotechnology and superintelligence, and their combination yields an even lower estimate, explaining much of the skepticism surrounding AI risk.

Therefore, it is reasonable to investigate catastrophic scenarios that do not rely on ideas with low prior probabilities.

No-miracle scenario where AI kill everybody

Imagine a possible world, where AI with IQ above 1000 is impossible and also nanotech doesn't work...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploratory Analysis of RLHF Transformers with TransformerLens, published by Curt Tigges on April 3, 2023 on The AI Alignment Forum.

TL;DR: I demonstrate how to use RLHF models trained with the TRLX library with TransformerLens, and how to take an exploratory look at how RLHF changes model internals. I also use activation patching to see what RLHF activations are sufficient to recreate some of the RLHF behavior in the source model. Note that this is simply a preliminary exploratory analysis, and much (much) work remains to be done. I hope to show that doing mechanistic interpretability analysis with RLHF models doesn't need to be intimidating and is quite approachable!

Introduction

LLMs trained with RLHF are a prominent paradigm in the current AI landscape, yet not much mechanistic interpretability work has been done on these models to date--partially due to the complexity and scale of these models, and partially due to the previous lack of accessible tooling for training and analysis.

Fortunately, we are reaching the point where tooling for both mechanistic interpretability and for RLHF fine-tuning is becoming available. In this blog post, I demonstrate how to do both RLHF training using TRLX, an open-source library created by CarperAI; and mechanistic interpretation of TRLX models using TransformerLens, a library created by Neel Nanda. Rather than going deep into specific findings, I want to illustrate some processes and tools I think are useful.

This post is intended to summarize and go alongside an interactive Colab; you can find that here.

I first fine-tune a movie-review-generating version of GPT-2 with TRLX to generate only negatively-biased movie reviews, following an example provided in the TRLX repo. I then load and analyze the model (and the original model before RLHF) into TransformerLens for mechanistic interpretability analysis. Here, I adapt some of the techniques and code from Neel Nanda's excellent Exploratory Analysis Demo.

In addition to carrying out some basic analysis to understand how different layers contribute to the logits, I also identify some key regions of the network responsible for contributing the negative bias to the network (at least, for the specific task of predicting the next adjective). Much analysis remains to be done, but I hope this work provides a useful starting point.

Importance of RLHF

RLHF (or sometimes, RLAIF, or RL from AI Feedback) is becoming increasingly important as a method for specifying the behavior of LLMs like OpenAI's ChatGPT or Anthropic's Claude. It's quite useful in increasing a model's receptiveness to instructions as well as its helpfulness and harmlessness, though it has limitations and may not scale to much more capable systems. Nevertheless, it is quite important in today's LLM landscape.

RL induces behavior in models that are critical to understand as we delegate more tasks to them. Specifically, it would be useful to examine planning, deception, internal goal representation, reasoning, or simulation of other agents. Neel Nanda provides a set of recommended RL problems in his 200 Open Problems in Mechanistic Interpretability sequence. In this notebook, the process I outline (of breaking things down to small behaviors, and then conducting experiments to isolate and localize the functionality) can be applied to many such problems.

RLHF Training Details

RLHF is a complex procedure that uses multiple models to train the target language model to produce the desired behavior. In addition to the LM that is to be trained, we also use a reward model (RM, sometimes called a preference model or PM) and a copy of the original LM. The process is as follows:

We first train a reward model on human preference data. The RM is usually just another language model to which we append an additional linear layer that will re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ultimate ends may be easily hidable behind convergent subgoals, published by TsviBT on April 2, 2023 on LessWrong.

[Metadata: crossposted from. First completed December 18, 2022.]

Thought and action in pursuit of convergent instrumental subgoals do not automatically reveal why those subgoals are being pursued--towards what supergoals--because many other agents with different supergoals would also pursue those subgoals, maybe with overlapping thought and action. In particular, an agent's ultimate ends don't have to be revealed by its pursuit of convergent subgoals. It might might therefore be easy to covertly pursue some ultimate goal by mostly pursuing generally useful subgoals of other supergoals. By the inspection paradox for the convergence of subgoals, it might be easy to think and act almost comprehensively like a non-threatening agent would think and act, while going most of the way towards achieving some other more ambitious goal.

Note: the summary above is the basic idea. The rest of the essay analyzes the idea in a lot of detail. The final main section might be the most interesting.

What can you tell about an agent's ultimate intent by its behavior?

An agent's ultimate intent is what the agent would do if it had unlimited ability to influence the world. What can we tell about an agent's ultimate intent by watching the external actions it takes, whether low-level (e.g. muscle movements) or higher-level (e.g. going to the store), and by watching its thinking (e.g. which numbers it's multiplying, which questions it's asking, which web searches it's running, which concepts are active)?

Terms

The cosmos is everything, including the observable world, logical facts, observers and agents themselves, and possibilities.

A goal-state is a state of the cosmos that an agent could try to bring about. (A state is equivalent to a set of states; more formally, the cosmos is a locale of possibilities and a goal-state is an open subspace.)

An instantiation of a cosmos-state is a substate. E.g. [I have this particular piece of gold] is an instantiation of [I have a piece of gold in general].

A goal is a property of an agent of the form: this agent is trying to bring about goal-state G. Overloading the notation, we also say the agent has the goal G, and we can imprecisely let the term also stand for whatever mental elements constitute the goal and its relations to the rest of the agent.

A pursuit of G is behavior, a set of actions, selected to achieve G. Since an agent having a goal G is basically the same as the agent pursuing G, we also say e.g. "the farmer has the goal [grow crops]", which really means, "the farmer selects actions that ze expects will bring about zer goal state [have crops], and so ze behaves in a way that's well-described as [growing crops]".

A strategy (for G) is a pursuit (of G) that involves an arrangement of goals (perhaps in sequence or in multiple parallel lines, perhaps with branching on contingencies).

If an agent is following a strategy for G, then for each goal Gs in the strategy, Gs is a subgoal of the agent's goal of G, and G is a supergoal of Gs. We can also loosely say that some goal G′ is a subgoal of G if G′ helps achieve G, whether or not there's an agent pursuing G using G′, meaning vaguely something like "there's a strategy sufficient to achieve G that uses G′ as a necessary part" or "many agents will do reasonably well in pursuing G if they use G′".

An intermediate goal is a goal that motivates the pursuit of subgoals, and itself is motivated by the pursuit of one or more supergoals.

The motivator set of a goal G is the set of goals Gs such that Gs is plausibly a supergoal of G.

The (sufficient) strategy set of a goal G is the set of strategies that are sufficient to achieve G.

Inferring supergoals through subgoals

Suppose that G is an in...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exposure to Lizardman is Lethal, published by Duncan Sabien on April 2, 2023 on LessWrong.

This is a short, experimentally-off-the-cuff post about one way in which social groups and institutions are effectively destroyed, in practice. It's fairly straightforward, but I don't try to connect all the dots (or even name all of them).

In a recent FB post, I made the following (lightly edited) claims:

I think a pretty solid heuristic is to treat 5% as 0% on things involving large numbers of people.

This is often referred to as "lizardman's constant;" my own first exposure to the idea was the fact that there are surveys in which ~5% of Obama voters report thinking Obama is the antichrist.

What's that 5% made of? WHO KNOWS. Human error, lulzy trolls, actual crazies, people having a really bad day, people who have complex reasons for the choices they make—the list of possible explanations goes on and on and probably all of them are in the mix, to some extent.

But like. This is always the case. If you're dealing with substantially-more-than-Dunbar's-number of people, I think you need to expect around 5% of your data to just be frothing insanity, total contentless noise.

If you're studying some awful thing X, and you get your approval rating/advocates of X down to 5% of the population, this is effectively zero. It's not gonna go lower without some kind of filter; you're dealing with quantum chaos at that point.

One of the things that institutions and authority do is provide insulation from this frothing madness.

e.g. you have a deranged suburban woman wanting a black man in the neighborhood to be arrested, for no discernible reason besides that he is black and she is crazy. In the best cases, the police dispatcher who takes the call recognizes that there is no real situation, and doesn't send an officer; in the second-best cases, the officer arrives on scene, assesses the non-situation, and defuses things by informing the crazy person that they are being crazy and that Authority does not deign to take action.

(There are much worse cases, of course.)

Similarly, a deranged parent calls up a school superintendent wanting a principal to be fired because their child was exposed to Michaelangelo's David, and the superintendent laughs and gently communicates "No, we are not doing that."

A key feature of this kind of insulation is that the person (or group, or structure) under attack, and vulnerable to attack, is different from the person (or group, or structure) doing the defending/dismissing. The defender/dismisser/insulator needs to be not vulnerable to the disapprobation of the lizardman—a superintendent who is not worried about losing his job, or a police officer who knows that his superior officer has his back. This was the original reasoning behind judges-elected-for-life—that society needed principled men and women of discernment who did not need to placate or cater to lizardman.

(Yes, there are ways that this can backfire and metastasize; I'm not saying that all such insulations are good but I am saying that all the good insulations have this property.)

Here's what happens, absent that insulation:

There's no intermediary here—no single sane person who feels personally unthreatened who is willing to say "what? No. We're not banning them from performing this song; that's ridiculous; the song is fine, the objections of lizardman notwithstanding."

Social media has given lizardman power and reach and concentration of force; it's harder to tune out lizardman, harder to insulate oneself from him, harder to simply close down the conversation and make a final call, the way that courts close down the conversation and make a final call in matters of justice.

(Even where the final call is wrong some non-negligible percentage of the time, it's still vastly better, from a population perspec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ultimate ends may be easily hidable behind convergent subgoals, published by Tsvi Benson-Tilsen on April 2, 2023 on The AI Alignment Forum.

[Metadata: crossposted from. First completed December 18, 2022.]

Thought and action in pursuit of convergent instrumental subgoals do not automatically reveal why those subgoals are being pursued--towards what supergoals--because many other agents with different supergoals would also pursue those subgoals, maybe with overlapping thought and action. In particular, an agent's ultimate ends don't have to be revealed by its pursuit of convergent subgoals. It might might therefore be easy to covertly pursue some ultimate goal by mostly pursuing generally useful subgoals of other supergoals. By the inspection paradox for the convergence of subgoals, it might be easy to think and act almost comprehensively like a non-threatening agent would think and act, while going most of the way towards achieving some other more ambitious goal.

Note: the summary above is the basic idea. The rest of the essay analyzes the idea in a lot of detail. The final main section might be the most interesting.

What can you tell about an agent's ultimate intent by its behavior?

An agent's ultimate intent is what the agent would do if it had unlimited ability to influence the world. What can we tell about an agent's ultimate intent by watching the external actions it takes, whether low-level (e.g. muscle movements) or higher-level (e.g. going to the store), and by watching its thinking (e.g. which numbers it's multiplying, which questions it's asking, which web searches it's running, which concepts are active)?

Terms

The cosmos is everything, including the observable world, logical facts, observers and agents themselves, and possibilities.

A goal-state is a state of the cosmos that an agent could try to bring about. (A state is equivalent to a set of states; more formally, the cosmos is a locale of possibilities and a goal-state is an open subspace.)

An instantiation of a cosmos-state is a substate. E.g. [I have this particular piece of gold] is an instantiation of [I have a piece of gold in general].

A goal is a property of an agent of the form: this agent is trying to bring about goal-state G. Overloading the notation, we also say the agent has the goal G, and we can imprecisely let the term also stand for whatever mental elements constitute the goal and its relations to the rest of the agent.

A pursuit of G is behavior, a set of actions, selected to achieve G. Since an agent having a goal G is basically the same as the agent pursuing G, we also say e.g. "the farmer has the goal [grow crops]", which really means, "the farmer selects actions that ze expects will bring about zer goal state [have crops], and so ze behaves in a way that's well-described as [growing crops]".

A strategy (for G) is a pursuit (of G) that involves an arrangement of goals (perhaps in sequence or in multiple parallel lines, perhaps with branching on contingencies).

If an agent is following a strategy for G, then for each goal Gs in the strategy, Gs is a subgoal of the agent's goal of G, and G is a supergoal of Gs. We can also loosely say that some goal G′ is a subgoal of G if G′ helps achieve G, whether or not there's an agent pursuing G using G′, meaning vaguely something like "there's a strategy sufficient to achieve G that uses G′ as a necessary part" or "many agents will do reasonably well in pursuing G if they use G′".

An intermediate goal is a goal that motivates the pursuit of subgoals, and itself is motivated by the pursuit of one or more supergoals.

The motivator set of a goal G is the set of goals Gs such that Gs is plausibly a supergoal of G.

The (sufficient) strategy set of a goal G is the set of strategies that are sufficient to achieve G.

Inferring supergoals through subgoal...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing: Effective Tourism - Maximizing the Cost-Effectiveness of Your Vacations, published by Agustín Covarrubias on April 1, 2023 on The Effective Altruism Forum.

TL;DR

After several months of research, EA Chile is proud to announce its work in a new cause area: Effective Tourism. This program aims to help EAs find cost-effective vacations that uniquely maximize their interests. Alongside our university group, we're also launching a 7-week introductory program, where participants will learn how to evaluate different vacation destinations using one of our main developments, the RTN framework.

Introduction

Want a break from all the recent developments in artificial intelligence? Feel unable to keep up with all the community controversies? Tired of dealing with mosquitoes?

We hear you! That's why we are excited to announce a new project by EA Chile: Effective Tourism. This new cause area is focused on finding the most cost-effective vacations for EAs, so you can recharge your batteries while still maximizing your impact.

We believe that a well-rested, recharged, and inspired EA community is better equipped to tackle the challenges of AI safety, pandemic prevention, global health and development, and much, much more. That's why we've decided to focus our efforts on this new initiative, to help EAs make the most of their time off, all while keeping cost-effectiveness in mind.

The RTN Framework

To prioritize between destinations, we've developed the Relaxation-Tractability-Neglectedness (RTN) framework, inspired by the Importance-Tractability-Neglectedness framework used in Effective Altruism. The RTN framework assesses vacation destinations using three key criteria:

Relaxation Potential: Evaluates the ability of a destination to provide a rejuvenating experience, considering factors like natural beauty, tranquility, recreational activities, and opportunities for mental and physical relaxation.

Tractability: Measures the ease of planning, organizing, and executing a trip to a specific destination, taking into account accessibility, affordability, ease of obtaining visas, and the availability of amenities and services.

Neglectedness: Refers to the degree to which a destination is overlooked or underappreciated by the broader tourist industry. By choosing neglected destinations, EAs can minimize the negative environmental and social impacts of tourism while enjoying unique, less crowded experiences.

Applying the RTN framework allows EAs to make informed decisions about where to spend their time off, ensuring cost-effective, rejuvenating vacations that align with their values and minimize the negative consequences of tourism.

The Introductory Program

In collaboration with our student group, Effective Altruism @ UC Chile, we're also launching an “Introduction to Effective Tourism” program, akin to the Introduction to Effective Altruism program. This program consists of 7 weeks of group discussions with a facilitator, focused on exploring different vacation destinations and finding out which is the most cost-effective for EAs.

Conclusion

We believe that the introduction of Effective Tourism as a new cause area has the potential to greatly benefit the EA community. By finding the most cost-effective vacations for EAs, we can help ensure that our community remains motivated, focused, and dedicated to our broader goals. We hope that you'll join us in our journey towards better, more effective vacations!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ending Open Philanthropy Project, published by Dusten Muskovitz on April 2, 2023 on The Effective Altruism Forum.

Effective immediately, my wife and I will no longer plan funding for EA or EAs. There’s enough money with OpenPhil to wind down operations gracefully, paying out all current grants, all grants under consideration that we normally would have made, and any new grants that come in within the next three months that we normally would have said yes to (existing charities receiving Open Philanthropy money are particularly encouraged to apply), and providing six months for everyone currently at the non-profit before we shut down.

I want to emphasize that this is not because of anything that Alexander Berger or the rest of the wonderful team at OpenPhil have done. They’re great, and I think that they’ve tried as hard as anyone could to do the best possible work with our money.

It’s the rest of you. I present three primary motivations. They're all somewhat interrelated, but hopefully by presenting three arguments in succession you will update on each of them sequentially. Certainly I've lost all hope in y'all retaining any of the virtues of the rationalist community, rather than just their vices. I hope that this helps you as a community clean up your act while you try to convince someone else to fund this mess. Maybe Bernauld Arnault. That was a joke. Haha, fat chance.

1. In the words of philosopher Liam Kofi Bright, “Why can’t you just be normal?”

Two of Redwood's leadership team have or have had relationships to an [Open Philanthropy] grant maker. A Redwood board member is married to a different OP grantmaker. A co-CEO of OP is one of the other three board members of Redwood. Additionally, many OP staff work out of Constellation, the office that Redwood runs. OP pays Redwood for use of the space.

Just be normal. Stop having a social community where people live and work and study and sing together, and do social atomization like everybody else. This won’t cause any problems. Everyone else is doing it. There is another way! You don't, actually, need to have more partners than the average American adult has friends.

Also, just don’t have sex. That’s not that much to ask for, is it? I’ve been married for a decade now: I can tell you, it’s perfectly possible.

2. I’m tired of all the criticism. I’m tired of it hitting Asana, which I still love and care about. Moving my donations to instead buying superyachts, artwork, and expanding on an actually fun hobby (giant bonsai) is going to substantially reduce how often my family, friends, and employees sees me getting attacked in one news outlet or another.

3. Pick a cause and stick with it. Have the courage of your convictions. I don't need to spend my time hearing about sea-rats and prisoners and suffering matrices and matrices that are suffering and discount rates and so many different ways human bodies can go wrong in other countries and immigration and housing for techies and so many more. Y'all were supposed to be optimizers, so this donating splitting between different cause areas should end. Like I said, most of my wealth is going into the new super-yacht my wife and I will be commissioning. Maybe then you could stop arguing quite so much. Get it all out of your systems, figure out what the best charity is, and stick with it.

Average American adult has three or fewer friends.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI community building: EliezerKart, published by Christopher King on April 1, 2023 on LessWrong.

Having good relations between the various factions of AI research is key to achieving our common goal of a good future. Therefore, I proposal an event to help bring us all together: EliezerKart! It is a go karting competition between three factions: AI capabilities researchers, AI existential safety researchers, and AI bias and ethics researchers.

The word Eliezer means "Help of my God" in Hebrew. The idea is whichever team is the best will have the help of their worldview, "their god", during the competition. There is no relation to anyone named Eliezer whatsoever.

The race will probably take place in the desert or some cool city or something.

Factions

Here is a breakdown of the three factions:

Capabilities

They are the most straight forward faction, but also the most technical. They can use advanced AI to create go kart autopilot, can simulate millions of races courses in advance to create the perfect cart, and can use GPT to couch their drivers. Unfortunately, they are not good at getting things right on the first critical try.

Safety

Safety has two overlapping subfactions.

Rationalists

Rationalists can use conditional prediction markets (kind of like a Futarchy) and other forecasting techniques to determine the best drivers, the best learning methods, etc... They can also use rationality to debate go kart driving technique much more rationally than the other factions.

Effective Altruists

The richest faction, they can pay for the most advanced go karts. However, they will spend months debating the metrics upon which to rate how "advanced" a go kart is.

Safety also knows how to do interpretability, which can create adversarial examples to throw off capabilities.

Bias and ethics

The trickiest faction, they can lobby the government to change the laws and the rules of the event ahead of time, or even mid-race. They can also turn the crowd against their competitors. They can also refuse to acknowledge the power of the AI used by capabilities altogether; whether their AI will care remains to be seen.

Stakes

Ah, but this isn't simply a team building exercise. There are also "prizes" in this race. Think of it kind of like a high stakes donor lottery.

If capabilities wins:

The other factions can not comment on machine learning unless they spend a week trying to train GANs.

Safety must inform capabilities of any ideas they have that can help create an even more helpful, harmless, and most importantly profitable assistant.

Bias and ethics must join the "safety and PR" departments of the AI companies.

If safety wins:

Everyone gets to enjoy a nice long AI summer!

Capabilities must spend a third of their time on interpretability and another third on AI approaches that are not just big inscrutable arrays of numbers.

Bias and ethics must only do research on if AI is biased towards paperclips, and their ethics teams must start working for the effective altruists, particularly on the "is everyone dying ethical?" question.

Bias and ethics must lobby the government to air strike all the GPU data centers.

If bias and ethics win:

Every capabilities researcher will have a bias and ethics expert sit behind them while they work. Anytime the capabilities researcher does something just because they can, the bias and ethics expert whispers technology is never neutral and the capabilities researcher's car is replaced by one that is 10% cheaper.

AI safety researchers must convert from their Machine God religion to atheism. They must also commit to working on an alignment strategy that, instead of maximizing CEV, minimizes the number of naughty words in the universe.

Capabilities must create drones with facial recognition technology that follow the AI safety and AI capabilities factions around and s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Saving drowning children in light of perilous maximisation, published by calebp on April 2, 2023 on The Effective Altruism Forum.

Last year Holden Karnofsky wrote the post, “EA is about maximization, and maximization is perilous”. You could read the post, but I suggest you just jump on board because Holden is cool, and morality is hard.

Given that you now believe that maximisation of doing good is actually bad and scary, you should also probably make some adjustments to the classic thought experiment you use to get your friends on board with the new mission of “do the most good possible [a large but not too large amount of good] using evidence and reason”.

A slightly modified drowning child thought experiment goes as follows

Imagine that you are walking by a small pond, and you see five children drowning. You can easily save the child without putting yourself in great danger, but doing so will ruin your expensive shoes. Should you save the children?

Obviously, your first instinct is to save all the children. But remember, maximisation is perilous. It’s this kind of attitude that leads to atrocities like large financial crimes. Instead, you should just save three or four of the children. That is still a large amount of good, and importantly, it is not maximally large.

But what should you do if you encounter just one drowning child? The options at first pass seem bleak – you can either:

Ignore the child and let them drown (which many people believe is bad).

Save the child (but know that you have tried to maximise good in that situation).

I think there are a few neat solutions to get around these moral conundrums:

Save the child with some reasonable probability (say 80%).

Before wading into the shallow pond, whip out the D10 you were carrying in your backpack. If you roll an eight or lower, then go ahead and save the child. Otherwise, go about your day.

Only partially save the child

You may have an opportunity to help the child to various degrees. Rather than picking up the child and then ensuring that they find their parents or doing other previously thought as reasonable things, you could:

Move the child to shallower waters so they are only drowning a little bit.

Help the child out of the water but then abandon them somewhere within a 300m radius of the pond.

Create a manifold market on whether the child will be saved and bid against it to incentivise other people to help the child.

The QALY approach

Save the child but replace them with an adult who is not able to swim (but is likely to have fewer years of healthy life left).

Commit now to a policy of only saving children who are sufficiently old or likely to have only moderately healthy/happy lives.

The King Solomon approach

Cut the child in half and save the left half of them from drowning

Using these approaches, you should be able to convey the optimal most Holden-approved amount of good.

If you like, you can remember the heuristic “maximisation bad”.

As well as other things like eradicating diseases.

QALYs are quality-adjusted life years (essentially a metric for healthy years lived).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing AlignmentSearch: An AI Alignment-Informed Conversional Agent, published by BionicD0LPH1N on April 1, 2023 on LessWrong.

Authors: Henri Lemoine, Thomas Lemoine, and Fraser Lee

Note: Throughout our development process, we benchmarked with smaller scale vector embeddings, assuming performance would scale linearly. It appears that we were incorrect. It turns out the entirety of our augmented alignment database is too slow to be practically run on the servers we’re using without more specialized hardware, which we plan to migrate to over this weekend. As a stop gap, the live published version of AlignmentSearch is running with just over 11% of our embeddings - every piece of content in our dataset explicitly tagged with “AI”. From our testing, expect somewhat worse results than the full version.

We are excited to introduce AlignmentSearch, an attempt to create a conversational agent that can answer questions about AI alignment. We built this site in response to ArthurB’s $5k bounty for a LessWrong conversational agent calling for the creation of a chatbot capable of discussing bad AI Alignment takes, and guiding people new to the field through common misunderstandings and early points of confusion.

Tl;dr

AlignmentSearch uses a dataset about AI alignment to construct a prompt for ChatGPT to answer AI alignment-related questions, while citing established sources. We qualitatively observe a massive boost in the quality of answers over ChatGPT without any specialized prompt, with results either on par or better than those given by much stronger LLMs (GPT-4 and Bing Search).

Overview

AlignmentSearch indexes the Alignment Research Dataset, generating vector embeddings to enable nearest-semantic-neighbor search. We take a user query to find the top-k most semantically close “paragraphs” (text blocks of around 220 tokens, plus some padding). The size of the dataset means we generally have access to a few paragraphs that are very semantically similar to the user’s question, and are likely to contain information that is useful in producing an answer. We form a prompt that gives these paragraphs to ChatGPT along with the user’s question and instructions on citation, and retrieve an answer somewhere between summary and synthesis, all with accurate inline citations linking to the source material. We’ve created a website that lets the user interact easily with this process: ask questions, dive into the sources used in an answer, and ask for further clarifications. We also have an alternative mode that exposes the raw results of the semantic search.

Alignment Research Dataset

The alignment research dataset was announced on LessWrong in June 2022, with a related paper accessible here. It contains posts and papers from a wide range of websites and books about alignment, including LessWrong, stampy.ai, arXiv, the alignment newsletter, etc. As it was created in June ‘22, recent posts are absent. We are in the process of updating the dataset to contain recent AI alignment content.

Variable Source Quality

Not all sources are made equal. Posts on LessWrong with negative karma were removed, as to reduce the chances of the website parroting back foolishness. Moreover, stampy.ai contains a highly curated dataset of common question-answer pairs related to AI safety, and we plan on favoring this source in semantic searches. A more nuanced and optimization-driven prioritization system, a better curated dataset, and a better formatted prompt may further improve the quality of the answers.

Embeddings

We currently use OpenAI’s text-embedding-ada-002 embedding model. The motivation behind this decision comes from the recent price drop in OpenAI ada, now roughly 500x less expensive than its predecessor. Embedding the entire 1.2 GB dataset cost ~$50.

One might think embedding the entirety of L...