Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Hidden Complexity of Wishes - The Animation, published by Writer on September 28, 2023 on LessWrong.

This video introduces a series focused on the outer alignment problem. In future videos, we'll explore how this problem affects machine learning systems today and how it could lead to catastrophic outcomes for humanity.

The video is an adaptation of @Eliezer Yudkowsky's post The Hidden Complexity of Wishes. The text has been slightly changed, with Eliezer's approval, mainly because the original has many references to other articles.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Different views of alignment have different consequences for imperfect methods, published by Stuart Armstrong on September 28, 2023 on The AI Alignment Forum.

Almost any powerful AI, with almost any goal, will doom humanity. Hence alignment is often seen as a constraint on AI power: we must direct the AI's optimisation power in a very narrow direction. If the AI is weak, then imperfect methods of alignment might be sufficient. But as the AI's power rises, the alignment methods must be better and better. Alignment is thus a dam that has to be tall enough and sturdy enough. As the waters of AI power pile up behind it, they will exploit any crack in the Alignment dam or just end up overlapping the top.

So assume A is an Alignment method that works in environment E (where "environment" includes the physical setup, the AI's world-model and knowledge, and the AI's capabilities in general). Then we expect that there is an environment E' where A fails - when the AI's capabilities are no longer constrained by A.

Now, maybe there is a better alignment method A' that would work in E', but there's likely another environment E'' where A' fails. So unless A is "almost perfect", there will always be some environment when it will fail.

So we need A to be "almost perfect". Furthermore, in the conventional view, you can't get there by combining imperfect methods - "belt and braces" doesn't work. So if U is a utility function that partially defines human flourishing but is missing some key elements, and if B is a box that contains the AI so that it can only communicate with humans via text interfaces, then U+B is not much more of constraint than U and B individually. Most AIs that are smart enough to exploit U and break B, can get around U+B.

A consequence of that perspective is that imperfect methods are of little use for alignment. Since you can't get "almost perfect" by adding up imperfect methods, there's no point in developing imperfect methods.

Concept extrapolation/model splintering[1] has a different dynamic[2]. Here the key idea is to ensure that the alignment method extends safely across an environment change. So if the AI starts with alignment method A in environment E, and then moves to environment E', we must ensure that A transitions to A', where A' ensures that the AI doesn't misbehave in environment E'. Thus the key is some 'meta alignment' method MA that manages[3] the transition to the new environment[4].

An important difference with the standard alignment picture is that imperfect methods are not necessarily useless. If method MA1 helps manage the transition between E and E', it may also work for transitioning between E1,000,000 and E1,000,001. And "belts and braces" may work: even if neither MA1 nor MA2 work between E1,000,000 and E1,000,001, maybe MA1+MA2 can work.

As an example of the first phenomena, Bayes' theorem can be demonstrated in simple diagrams, but continues to be true in very subtle and complicated situations. The case of MA1+MA2 can be seen by considering them as subsets of human moral principles; then there are some situations where MA1 is enough to get an answer ("a good thing for two beings is worth twice as much as a good thing for only one being"), some where MA2 is enough ("the happiness of a human is worth 10 times that of a dog"), and some where both sets are needed ("compare the happiness of two humans versus 19 times that happiness for a dog").

So the requirements for successful contribution to concept extrapolation approaches are lower than for more classical alignment methods: less perfection may be required.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From Passion to Depression and Pessimism: My Journey with Effective Altruism, published by IReallyWantToBeAnonymous on September 28, 2023 on The Effective Altruism Forum.

I want to share a deeply personal and painful journey I've had with the EA movement. It's not an easy story to tell, but I believe there's value in presenting this side of the coin. I really want to protect my anonymity, so I'd ask you to please be respectful of my wish and to not reach out to me.

Not so long ago, I became wholeheartedly committed to the EA cause. I left a good job after receiving funding to pursue work that resonated with the movement's principles. My belief was so strong that I relocated to another city, eager to make a meaningful impact. A lot of promises were made. A lot of enthusiasm surrounded EA future.

Then, the unexpected: my main source of funding collapsed. With its downfall, my life spiraled. I felt deserted by the very community I'd given so much to. Nobody reached out; nobody seemed to care. It was a profound isolation I had never anticipated.

This experience plunged me into a severe major depressive episode, one so grave I've grappled with all sorts of dark thoughts. I've now sought treatment for this, but every day is a struggle. For years, I sidelined personal pursuits, including forming meaningful personal and romantic relationships outside the movement, dedicating myself to issues like the potential AI apocalypse and other matters that now seem distant and abstract, when compared to the day-to-day struggles of non-Anglo-American privileged and gifted youngsters. In prioritizing these concerns, I lost sight of the spontaneous, daily realities that give life its texture and meaning.

My experience has also left me deeply disillusioned with EA's principles and strategies. I've become nihilistic, doubting if the movement's approach to the world, as noble as it might seem, is genuinely grounded in reality. There's a detachment I've observed, where some of the most crucial elements of our shared human experience, like the importance of spontaneous everyday moments, seem to get lost.

In sharing this, my hope isn't to condemn or vilify the EA movement but to highlight the dangers of over-commitment and the risk of losing oneself in a cause. While it's commendable to be passionate, it's essential to remember our humanity, the very thing we're trying to help and protect.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Petrov Day Retrospective, 2023 (re: the most important virtue of Petrov Day & unilaterally promoting it), published by Ruby on September 28, 2023 on LessWrong.

When Stanislav Petrov's missile alert system pinged, the world was not watching. Russia was not watching. Perhaps a number of superiors in the military were staying in the loop about Stanislav's outpost, waiting for updates. It wasn't theatre.

In contrast, LessWrong's historical Petrov Day celebrations have been pretty flashy affairs. Great big red buttons, intimidating countdown timers, and all that. That's probably not what the next "don't destroy the world" moment will look like.

It's also the case that some of the biggest moral dilemmas don't come clearly labeled as such, and don't have the options clearly marked as "cooperate" or "defect". (I think in Petrov's case, it was clear it was a big decision. Unclear to me how it easy it was for him to make and why.)

Matching the spirit of the above, this year's LessWrong commemoration was a little more one-on-one. It started with a poll. In previous year's, the LessWrong team has unilaterally decided the meaning of Petrov Day, often to objection. So why not get a sense of what people actually think matters most?

We sent the following private message to anyone who'd been active on LessWrong in the previous 24 hours:

252 people responded to the survey at the time I started work on this post, and the results are pretty clear:

The Most Important Value of Petrov Day

Note: We did not actually spend much time thinking about the options in this poll, their framing, etc. Like under 10 minutes. Feel free to discuss in the comments.

VirtueNumPercentAvoiding actions that noticeably increase the chance that civilization is destroyed14457%Accurately reporting your epistemic state2711%Quickly orienting to novel situations2510%Resisting social pressure5622%Total252100%

Results are not significantly different for users with 1000+ karma:

VirtueNumPercentAvoiding actions that noticeably increase the chance that civilization is destroyed3549%Accurately reporting your epistemic state811%Quickly orienting to novel situations811%Resisting social pressure5628%Total107100%

Unilaterally pushing your own values over the collective?

I don't know whether what really was going on was genuinely idealistic as opposed to symmetrical fighting over resources, but a lot of the USRussia conflict seemed to be about values and beliefs about what was right. Capitalism, communism, etc.

This raises some good questions. What are the legitimate ways to promote your own values over other people? This is where the follow-up poll question took us.

Users were divided on the most important virtue (we don't know their opinions on the other virtues listed re Petrov Day), but it seemed reasonable that next year we'd go with the majority (or at least plurality) as a focus.

However, part of the Petrov Day experience (imo) is individuals being options to unilaterally change how things go for everyone else. Such an option we did kindly provide.

After some discussion, the LessWrong team has decided to make the focus of next year's Petrov Day be the virtue that is selected as most important by the most people...If you click the below link and are the first to do so of any minority group, we will make your selected virtue be the focus of next year's commemoration [instead].

The plain value choice, according me, is faced with a values difference (or belief difference?) to go along with the majority, or to decide that unilaterally you'll take the opportunity to promote what you think is correct.

I find myself thinking about Three Worlds Collide scenarios where you come across others with different values, and possibly there are power differentials. What do you do confronted by baby eaters people who prioritize communicati...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Projects I would like to see (possibly at AI Safety Camp), published by Linda Linsefors on September 27, 2023 on The AI Alignment Forum.

I recently discussed with my AISC co-organiser Remmelt, some possible project ideas I would be excited about seeing at the upcoming AISC, and I thought these would be valuable to share more widely.

Thanks to Remmelt for helfull suggestions and comments.

What is AI Safety Camp?

AISC in its current form is primarily a structure to help people find collaborators. As a research lead we give your project visibility, and help you recruit a team. As a regular participant, we match you up with a project you can help with.

I want to see more good projects happening. I know there is a lot of unused talent wanting to help with AI safety. If you want to run one of these projects, it doesn't matter to me if you do it as part of AISC or independently, or as part of some other program. The purpose of this post is to highlight these projects as valuable things to do, and to let you know AISC can support you, if you think what we offer is helpful.

Project ideas

These are not my after-long-consideration top picks of most important things to do, just some things I think would be net positive if someone would do. I typically don't spend much cognitive effort on absolute rankings anyway, since I think personal fit is more important for ranking your personal options. I don't claim originality for anything here. It's possible there is work on one or several of these topics, which I'm not aware of. Please share links in comments, if you know of such work.

Is substrate-needs convergence inevitable for any autonomous system, or is it preventable with sufficient error correction techniques?

This can be done as an adversarial collaboration (see below) but doesn't have to be.

The risk from substrate-needs convergence can be summarised as such:

If AI is complex enough to self-sufficiently maintain its components, natural selection will sneak in.

This would select for components that cause environmental conditions needed for artificial self-replication.

An AGI will necessarily be complex enough.

Therefore natural selection will push the system towards self replication. Therefore it is not possible for an AGI to be stably aligned with any other goal. Note that this line of reasoning does not necessitate that the AI will come to represent self replication as its goal (although that is a possible outcome), only that natural selection will push it towards this behaviour.

I'm simplifying and skipping over a lot of steps! I don't think there currently is a great writeup of the full argument, but if you're interested you can read more here or watch this talk by Remmelt or reach out to me or Remmelt. Remmelt has a deeper understanding of the arguments for substrate-needs convergence than me, but my communication style might be better suited for some people.

I think substrate-needs convergence is pointing at a real risk. I don't know yet if the argument (which I summarised above) proves that building an AGI that stays aligned is impossible, or if it points to one more challenge to be overcome. Figuring out which of these is the case seems very important. I've talked to a few people about this problem, and identified what I think is the main crux: How well you can execute error correction mechanisms? When Forest Laundry and Anders Sandberg discussed substrate-needs convergence, they ended up with a similar crux, but unfortunately did not have time to address it. Here's a recording of their discussion, however Landry's mic breaks about 20 minutes in, which makes it hard to hear him from that point onward.

Any alignment-relevant adversarial collaboration

What is adversarial collaborations: [link to some Scott Post]

Possible topic:

For and against some alignment plan. Maybe y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Commonsense Good, Creative Good, published by Jeff Kaufman on September 27, 2023 on The Effective Altruism Forum.

Let's say you're vegan and you go to a vegan restaurant. The food is quite bad, and you'd normally leave a bad review, but now you're worried: what if your bad review leads people to go to non-vegan restaurants instead? Should you refrain from leaving a review? Or leave a false review, for the animals?

On the other hand, there are a lot of potential consequences of leaving a review beyond "it makes people less likely to eat at this particular restaurant, and they might eat at a non-vegan restaurant instead". For example, three plausible effects of artificially inflated reviews could be:

Non-vegans looking for high-quality food go to the restaurant, get vegan food, think "even highly rated vegan food is terrible", don't become vegan.

Actually good vegan restaurants have trouble distinguishing themselves, because "helpful" vegans rate everywhere five stars regardless of quality, and so the normal forces that push up the quality of food don't work as well. Now the food tastes bad and fewer people are willing to sustain the sacrifice of being vegan.

People notice this and think "if vegans are lying to us about how good the food is, are they also lying to us about the health impacts?" Overall trust in vegans (and utilitarians) decreases.

Despite thinking that it is the outcomes of actions that determine whether they are a good idea, I don't think this kind of reasoning about everyday things is actually helpful. It's too easy to tie yourself in logical knots, making a decision that seems counterintuitive-but-correct, except if you spent longer thinking about it, or discussed it with others, you would have decided the other way.

We are human beings making hundreds of decisions a day, with limited ability to know the impacts of our actions, and a worryingly strong capacity for self-serving reasoning. A full unbiased weighing of the possibilities is, sure, the correct choice if you relax these constraints, but in our daily lives that's not an option we have.

Luckily, humans have lived for centuries under these constraints, and we've developed ideas of what is "good" that turn out to be a solid guide to typical situations. Moral systems around the world don't agree on everything, but on questions of how to live your daily life they're surprisingly close: patience, respect, humility, moderation, kindness, honesty. I'm thankful we have all this learning on makes for harmonious societies distilled into our cultures to support us in our interactions.

On the other hand, I do think there is a very important place for this kind of reasoning: sometimes our normal ideas of "good" are seriously lacking. For example, they don't give us much guidance once scale is involved: a donation that helps a hundred people and one that equivalently helps a thousand people are both "good" from a commonsense perspective, even though I think it's pretty clearly ten times better to go with the second. Similarly, if you're trying to decide between working as a teacher in a poor school, therapist in a jail, a manager at a food pantry, or a firefighter in a disadvantaged community, common sense just says they're all "good" and leaves you there.

How do we reconcile this conflict, where carefully getting into the consequences of decisions can take a lot of time and risk strange errors, while never evaluating the outcomes of decisions risks having a much smaller positive effect on the world? I'd propose normally going for "commonsense good" and then in the most important cases going for "creative good".

The idea is, normally just do straightforwardly good things. Be cooperative, friendly, and considerate. Embrace the standard virtues. Don't stress about the global impacts or second-order altruisti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jacob on the Precipice, published by Richard Ngo on September 27, 2023 on LessWrong.

And he dreamed, and behold, there was a ladder set up on the earth, and the top of it reached to heaven. And behold, the angels of God were ascending and descending on it! And behold, the LORD stood above it and said, "I am the LORD, the God of Abraham your father and the God of Isaac. The land on which you lie I will give to you and to your offspring. Your offspring shall be like the dust of the earth, and you shall spread abroad to the west and to the east and to the north and to the south, and in you and your offspring shall all the families of the earth be blessed. Behold, I am with you and will keep you wherever you go, and will bring you back to this land. For I will not leave you until I have done what I have promised you." Then Jacob awoke from his sleep and said, "Surely the LORD is in this place, and I did not know it."

Genesis 28:12

That night Jacob arose and took his two wives, his two female servants, and his eleven children, and crossed the ford of the Jabbok. He sent them across the stream along with everything else that he had. And Jacob was left alone; and a man wrestled with him until the breaking of the day. When the man saw that he did not prevail against Jacob, he touched his hip socket, and Jacob's hip was put out of joint as he wrestled with him. Then the man said, "Let me go, for the day has broken." But Jacob said, "I will not let you go unless you bless me." And he said, "What is your name?" And he said, "Jacob." Then he said, "Your name shall no longer be called Jacob, but Israel, for you have striven with God and with men, and have prevailed."

Genesis 32:22

The ineffable is dead; science has killed it. Oh, there are still open questions, there are still things we don't know, but almost none of it is truly unimaginable any more. The origins of life: tide pools, maybe, or hydrothermal vents - we'll know once we can run more powerful simulations. Consciousness: looks like it's a pattern of recursive attention in a neural network, we'll be able to recreate it once we get better architecture searches working. Even the laws of physics themselves we can chalk up to the multiverse: if all possible universes exist, we can think of our own as just a random draw from the set of universes in which it's possible for life to flourish.

There's only one real mystery left. One thing that feels impossible to understand, even in principle: why here? Why have we found ourselves in this part of the multiverse, when there are so many other parts containing far more people? Why did I wake up as me, not them? Why am I living in the 21st century, balanced on a knife-edge, staring down ruin, instead of in a teeming glorious future?

It all came down to force versus momentum, in the end. Despite all our fancy technology, our god-like knowledge of the building blocks of the universe, only a single simple question ended up mattering: how much force can we apply, how fast, to deflect an asteroid how far?

There wasn't a single point where we found out that this was going to be the overriding purpose of our lives. But I remember where it started for me: at Andrea's watching party for the kickoff of the first big asteroid mining project. This was merely one of many steps, of course. Technically it didn't even involve any mining: they were just going to redirect the asteroid into orbit around Mars so that it'd be more accessible later on. But the metals from this asteroid would be used to set up factories on Mars to produce more asteroid-movers, which would be used to gather more resources, in a compounding spiral of astronomical-scale expansion. So it felt like a huge milestone: an unprecedented feat of human ingenuity, paving the way for our eventual conquest of the stars.

I'd met Andrea ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Our tofu book has launched!! (Upvote on Amazon), published by George Stiffman on September 27, 2023 on The Effective Altruism Forum.

As some of you know, I'm working on growing the US market for Chinese tofus. I believe it could be a way to significantly reduce animal suffering, while shifting American dining culture.

We just launched our book - Broken Cuisine - which introduces five of these tofus to Western home cooks. The goal is to spark curiosity and demand for these ingredients, so that we can convince retailers to carry them.

Do you have five minutes right now to take an action?

Download our FREE e-book on Amazon. (If possible, TODAY)

Skim through.

A day or two later, leave an honest review. (Amazon easily detects spam.)

Downloading and reviewing our book during launch week will convince the Amazon recommender algorithm to push our book, creating a virtuous cycle that will bring it to more people. If Broken Cuisine can crack the bestseller lists, I'm hopeful we can meaningfully start growing the market for these foods!

Thank you for your help!!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Bulwark's Article On Effective Altruism Is a Short Circuit, published by Omnizoid on September 27, 2023 on The Effective Altruism Forum.

Crosspost of this on my blog.

I really like the Bulwark. Their articles are consistently funny, well-written, and sensible. But recently, Mary Townsend wrote a critique of effective altruism titled "Effective Altruism Is a Short Circuit," that seems deeply confused. In fact, I will go further and make a stronger claim - there is not a single argument made in the entire article that should cause anyone to be less sympathetic to effective altruism at all! Every claim in the article is either true but irrelevant to the content of effective altruism or false. The article is crafted in many ways to mislead, confuse and induce negative affect in the reader but is light on anything of substance.

For instance, the article begins with a foreboding picture of the notorious EA fraudster Sam Bankman Fried. This is not an explicit argument given, of course - it's just a picture. If it were forced to be an argument, it would not succeed - even if Bernie Madoff gave a lot of money to the Red Cross and has some role in planning operations, that would do nothing to discredit the Red Cross; the same principle is true of EA. But when one is writing a smear piece, they don't need to include real objections - they can just include things that induce disdain in the reader that they come to associate with the object of the author's criticism. Such is reminiscent of the flashing red letters that are ubiquitous in attack ads - good if one's aim is propaganda, bad if one's aim is truth.

The article spends the first few paragraphs on mostly unremarkable philosophical musings about how we often have an urge to do good and we can choose what we do, filled with sophisticated-sounding references to philosophers and literature. Such musings help build the author's ethos as a Very Serious Person, but does little to provide an argument. However, after a few paragraphs of this, the author gets to the first real criticism:

That one could become good through monetary transactions should raise our post-Reformation suspicions, obviously. As a simple response to the stipulation of a dreadful but equally simple freedom, it seems almost designed to hit us at the weakest spots of our human frailty, with disconcerting effects.

Effective altruism doesn't claim, like those who endorsed indulgences, that one can become good through donating. It claims that one can do good through donating and that one should do good. The second half of that claim is a trivially obvious moral claim - we should help people more rather than less - and the first half of the claim is backed by quite overwhelming empirical evidence. While one can dispute the details somewhat, the claim that we can save the lives of faraway people for a few thousand dollars is incontrovertible given the weight of the available evidence - there's a reason that critics of EA never have specific criticisms of the empirical claims made by effective altruists.

Once one acknowledges that those who give to effective charities can save hundreds of lives over the course of their lives by fairly modest donations, a claim that even critics of such giving generally do not dispute, the claim that one should donate significant amounts to charities in order to save the lives of lots of people who would otherwise have died of horrifying diseases ceases to be something that "raises our post-reformation suspicions." One imagines the following dialogue (between Townsend and a starving child):

Child: Please, could I have five dollars. This would allow me to afford food today so I wouldn't be hungry.

Townsend: Sorry, I'd love to help, but that one could become good through monetary transactions should raise our post-Reformation suspici...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Prospect Magazine - How to save humanity from extinction, published by jackva on September 27, 2023 on The Effective Altruism Forum.

This is a story in Prospect Magazine featuring the work of several EAs (full disclosure: including myself) working on existential and catastrophic risks framed around the occasion of Petrov Day.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org