The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Christian Ruhl, senior global catastrophic risk researcher at Founders Pledge, published by Lizka on September 26, 2023 on The Effective Altruism Forum.

It's Petrov Day. One of the things we're doing to mark the occasion is hosting a thread where you're invited to ask Christian Ruhl anything.

Instructions:

Please post questions to Christian as comments on this post.

Sharing questions earlier is generally better; Christian will answer questions on Friday, September 29.

And you can upvote questions you're interested in.

Christian shared some context that might help draft questions (and you might be interested in exploring his posts!):

About me

I'm a senior researcher at Founders Pledge, where much of my work focuses on global catastrophic risks. Recently, I've written about philanthropy and nuclear security, including a long philanthropic guide on nuclear risks, an article in Vox with Longview's Matt Gentzel, "Call me, maybe?" about crisis communication, and on "philanthropy to the right of boom." I'm currently finishing up another "guide for philanthropists," this time focused on biosecurity and pandemic preparedness, which we'll publish later this fall. I'm also working on a new report about great power competition and transformative technologies with Stephen Clare and an investigation on germicidal UV with Rosie Bettle.

I've been at Founders Pledge for almost two years now. Before that, I worked at Perry World House, managing the ominous-sounding research theme on The Future of the Global Order: Power, Technology, and Governance. Before that, I studied two MPhil courses at Cambridge - History and Philosophy of Science and Politics and International Studies - funded by a Herchel Smith Fellowship. I first got interested in civilizational collapse and global catastrophic risks by working on a Maya archaeological excavation in Guatemala.

Question topics

I'm happy to talk about anything, including the sorry state of nuclear security philanthropy, working at Founders Pledge, working at an academic think tank, research, writing, civilizational collapse, global catastrophic and existential risks, great power competition, and more.

Other notes

If you want to help support projects to mitigate global catastrophic risks, please consider donating to the GCR Fund via every.org and Giving What We Can (or if you're a Founders Pledge member, from your DAF through the member app).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The King and the Golem, published by Richard Ngo on September 26, 2023 on LessWrong.

Long ago, there was a mighty king who had everything in the world that he wanted, except trust. Who could he trust, when anyone around him might scheme for his throne? So he resolved to study the nature of trust, that he might figure out how to gain it. He asked his subjects to bring him the most trustworthy thing in the kingdom, promising great riches if they succeeded.

Soon, the first of them arrived at his palace to try. A teacher brought her book of lessons. "We cannot know the future," she said, "But we know mathematics and chemistry and history; those we can trust." A farmer brought his plow. "I know it like the back of my hand; how it rolls, and how it turns, and every detail of it, enough that I can trust it fully."

The king asked his wisest scholars if the teacher spoke true. But as they read her book, each pointed out new errors - it was only written by humans, after all. Then the king told the farmer to plow the fields near the palace. But he was not used to plowing fields as rich as these, and his trusty plow would often sink too far into the soil. So the king was not satisfied, and sent his message even further afield.

A merchant brought a sick old beggar. "I met him on the road here, and offered him food, water, and shelter. He has no family, and only a short time left to live, during which I will provide for his every need. He has nothing to gain from betraying me; this is what allows true trust." A mother brought her young daughter. "I've raised her to lack any evil in her heart, to say only good words and do only good deeds. As long as she is not corrupted, she will remain the most trustworthy in the kingdom."

The king asked the beggar, "How did you end up in such dire straits?" The beggar let out a sigh, and recounted his sorrows: the neighbors who refused to help him when his crops failed; the murder of his son by bandits as they traveled to a new town; the sickness that took his wife as she labored for a pittance in squalid conditions.

"So you have been wronged?" the king asked.

"Very surely", the beggar said.

"I will give you revenge on the ones who have wronged you, then. All I ask is for you to denounce this merchant." The beggar's decision did not take long - for the trust that came easily was broken easily too.

To the mother, the king asked: "How did you raise such a child? Has she never once strayed?"

"Well, once or twice. But I discipline her firmly, and she learns fast."

The king, who knew something of children, ruled that for a month nobody would discipline the child in any way. By the end of it, she was as wild and tempestuous as any in the palace. So the king remained unsatisfied, and renewed his call for the most trustworthy thing in the kingdom.

Now his subjects became more creative. An economist brought him a book of statistical tables. "Any individual might vary and change," he said, "but in aggregate, their behavior follows laws which can be trusted." A philosopher brought a mirror. "By your own standards only you are truly trustworthy, sire; nothing else can compare."

The king scrutinized the economist's tables. "The trend changed here, fifteen years ago" he said, pointing. "Why?" The economist launched into a long, complicated explanation.

"And did you discover this explanation before or after it happened?" the king asked.

The economist coughed. "After, your highness."

"If you tell me when the next such change will happen, I will bestow upon you great rewards if you are right, but great penalties if you are wrong. What say you?" The economist consulted his books and tables, but could not find what he sought there, and left court that same night.

As for the philosopher, the king ordered him whipped. The philosopher protested: it would be an unjust...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New page on animal welfare on Our World in Data, published by EdMathieu on September 25, 2023 on The Effective Altruism Forum.

Our team at Our World in Data just launched a new page on animal welfare! There you can find a brand new Animal Welfare Data Explorer, 22 interactive charts, and 4 new articles:

How many animals get slaughtered every day?

How many animals are factory-farmed?

Do better cages or cage-free environments really improve the lives of hens?

Adopting slower-growing breeds of chicken would reduce animal suffering significantly

On Our World in Data, we cover many topics related to reducing human suffering: alleviating poverty, reducing child and maternal mortality, curing diseases, and ending hunger.

But if we aim to reduce total suffering, society's ability to reduce this in other animals - which feel pain, too - also matters.

This is especially true when we look at the numbers: every year, humans slaughter more than 80 billion land-based animals for farming alone. Most of these animals are raised in factory farms, often in painful and inhumane conditions.

Estimates for fish are more uncertain, but when we include them, these numbers more than double.

These numbers are large - but this also means that there are large opportunities to alleviate animal suffering by reducing the number of animals we use for food, science, cosmetics, and other industries and improving the living conditions of those we continue to raise.

On this page, you can find all of our data, visualizations, and writing on animal welfare.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Impact stories for model internals: an exercise for interpretability researchers, published by Jenny Nitishinskaya on September 25, 2023 on The AI Alignment Forum.

Inspired by Neel's longlist; thanks to @Nicholas Goldowsky-Dill and @Sam Marks for feedback and discussion, and thanks to AWAIR attendees for participating in the associated activity.

As part of the Alignment Workshop for AI Researchers in July/August '23, I ran a session on theories of impact for model internals. Many of the attendees were excited about this area of work, and we wanted an exercise to help them think through what exactly they were aiming for and why. This write-up came out of planning for the session, though I didn't use all this content verbatim. My main goal was to find concrete starting points for discussion, which

have the right shape to be a theory of impact

are divided up in a way that feels natural

cover the diverse reasons why people may be excited about model internals work

(according to me).

This isn't an endorsement of any of these, or of model internals research in general. The ideas on this list are due to many people, and I cite things sporadically when I think it adds useful context: feel free to suggest additional citations if you think it would help clarify what I'm referring to.

Summary of the activity

During the session, participants identified which impact stories seemed most exciting to them. We discussed why they felt excited, what success might look like concretely, how it might fail, what other ideas are related, etc. for a couple of those items. I think categorizing existing work based on its theory of impact could also be a good exercise in the future.

I personally found the discussion useful for helping me understand what motivated some of the researchers I talked to. I was surprised by the diversity.

Key stats of an impact story

Applications of model internals vary a lot along multiple axes:

Level of human understanding needed for the application

If a lot of human understanding is needed, does that update you on the difficulty of executing in this direction? If understanding is not needed, does that open up possibilities for non-understanding-based methods you hadn't considered?

For example, determining whether the model does planning would probably require understanding. On the other hand, finding adversarial examples or eliciting latent knowledge might not involve any.

Level of rigor or completeness (in terms of % model explained) needed for the application

If a high level of rigor or completeness is needed, does that update you on the difficulty of executing in this direction? What does the path to high rigor/completeness look like? Can you think of modifications to the impact story that might make partial progress be more useful?

For example, we get value out of finding adversarial examples or dangerous capabilities, even if the way we find them is somewhat hacky. Meanwhile, if we don't find them, we'd need to be extremely thorough to be sure they don't exist, or sufficiently rigorous to get a useful bound on how likely the model is to be dangerous.

Is using model internals essential for the application, or are there many possible approaches to the application, only some of which make use of model internals?

Steering model behaviors can be done via model editing, or by prompting or finetuning; but, there are reasons (mentioned below) why editing could be a better approach.

Many impact stories (at least as I've categorized them) have variants that live at multiple points on these spectra. When thinking about one, you should think about where it lands, and what variants you can think of that might be e.g. easier but still useful.

The list

Some are more fleshed out than others; some of the rest could be fleshed out with a bit more effort, while others are more ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is wrong with this "utility switch button problem" approach?, published by Donald Hobson on September 25, 2023 on The AI Alignment Forum.

Suppose you have some positive utility functions A,B , mathematically considered to be random variables dependant on choice of policy.

Let K be a random variable that is 1 if the button is pressed, 0 otherwise. Again dependant on choice of policy. I am considering that there is a particular time that the button must be pressed to make this a single bit. You have a single chance to switch this AI between 2 utilities, and whatever you pick in that instant is baked in forevermore.

Let a=E(A(1-K)), b=E(BK) be the expected partial utilities.

Now consider all policies. In particular, consider the pareto frontier between a and b. Through the introduction of randomness, we can make that parito frontier continuous.

We want some policy off that frontier.

There will be some policy ha that throws all resources at maximizing a. And there will be some (usually different) policy hb that throws all available resources at maximizing b.

let q be a tradeoff rate. And u=a(1-q)+bq. The parito frontier can now be defined as the policies that maximize u for varying q.

For ha, this is the policy that is optimal when q=0 which has u(ha)=a(ha). Then dudE(K)<0.

Likewise at hb with q=1 have dudE(K)>0.

So somewhere in the middle, by the continuity granted by stochastic mixes of policies, their must be at least one point where dudE(K)=0. Use that policy, or stochastic mixture of policies.

This agent will never pay ϵ utility to set the value of the button one way or the other. Because the policy is one in which has E(u|K=0)=E(u|K=1) (I think) Which hopefully means that this policy does worse than the policy that doesn't pay to change the button, but otherwise does everything else the same.

Put another way, if you pay to flip the button. Then you must care about the state of the button, if the chance the button is pressed changes, you are getting more or less for your ϵ utility. So the gradient can't be 0.

And this is described as a choice of policy. A framework which automatically includes the actions of all sub agents.

So, how does this fail? Who can break this agent and get it to do crazy things?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Amazon to invest up to $4 billion in Anthropic, published by Davis Kingsley on September 25, 2023 on The Effective Altruism Forum.

Today, we're announcing that Amazon will invest up to $4 billion in Anthropic. The agreement is part of a broader collaboration to develop reliable and high-performing foundation models.

(Thread continues from there with more details -- seems like a notable major development!)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "X distracts from Y" as a thinly-disguised fight over group status / politics, published by Steven Byrnes on September 25, 2023 on LessWrong.

1. Introduction

There's a popular argument that says:

It's bad to talk about whether future AI algorithms might cause human extinction, because that would be a distraction from the fact that current AI algorithms are right now causing or exacerbating societal problems (misinformation, deepfakes, political polarization, algorithmic bias, maybe job losses, etc.)

For example, Melanie Mitchell makes this argument (link & my reply here), as does Blake Richards (link & my reply here), as does Daron Acemoglu (link & a reply by Scott Alexander here & here), and many more.

In Section 2 I will argue that if we try to flesh out this argument in the most literal and straightforward way, it makes no sense, and is inconsistent with everything else these people are saying and doing. Then in Section 3 I'll propose an alternative elaboration that I think is a better fit.

I'll close in Section 4 with two ideas for what we can do to make this problem better.

(By "we", I mean "people like me who are very concerned about future AI extinction risk (x-risk)". That's my main intended audience for this piece, although everyone else is welcome to listen in too. If you're interested in why someone might believe that future AI poses an x-risk in the first place, you're in the wrong place - try here or here.)

2. Wrong way to flesh out this argument: This is about zero-sum attention, zero-sum advocacy, zero-sum budgeting, etc.

If we take the "distraction" claim above at face value, maybe we could flesh it out as follows:

Newspapers can only have so many front-page headlines per day. Lawmakers can only pass so many laws per year. Tweens can only watch so many dozens of TikTok videos per second. In general, there is a finite supply of attention, time, and money. Therefore, if more attention, time, and money is flowing to Cause A (= future AI x-risk), then that means there's less attention, time and money left over for any other Cause B (= immediate AI problems).

I claim that this is not the type of claim that people are making. After all, if that's the logic, then the following would be equally sensible:

"It's bad to talk about police incompetence, because it's a distraction from talking about police corruption."

"It's bad to talk about health care reform, because it's a distraction from talking about climate change."

Obviously, nobody makes those arguments. (Well, almost nobody - see next subsection.)

Take the first one. I think it's common sense that concerns about police incompetence do not distract from concerns about police corruption. After all, why would they? It's not like newspapers have decided a priori that there will be one and only one headline per month about police problems, and therefore police incompetence and police corruption need to duke it out over that one slot. If anything, it's the opposite! If police incompetence headlines are getting clicks, we're likely to see more headlines on police corruption, not fewer. It's true that the total number of headlines is fixed, but it's perfectly possible for police-related articles to collectively increase, at the expense of articles about totally unrelated topics like Ozempic or real estate.

By the same token, there is no good reason that concerns about future AI causing human extinction should be a distraction from concerns about current AI:

At worst, they're two different topics, akin to the silly idea above that talking about health care reform is a problematic distraction from talking about climate change.

At best, they are complementary, and thus akin to the even sillier idea above that talking about police corruption is a problematic distraction from talking about police incompetence.

Suppor...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inside Views, Impostor Syndrome, and the Great LARP, published by johnswentworth on September 25, 2023 on LessWrong.

Epistemic status: model which I find sometimes useful, and which emphasizes some true things about many parts of the world which common alternative models overlook. Probably not correct in full generality.

Consider Yoshua Bengio, one of the people who won a Turing Award for deep learning research. Looking at his work, he clearly "knows what he's doing". He doesn't know what the answers will be in advance, but he has some models of what the key questions are, what the key barriers are, and at least some hand-wavy pseudo-models of how things work.

For instance, Bengio et al's "Unitary Evolution Recurrent Neural Networks". This is the sort of thing which one naturally ends up investigating, when thinking about how to better avoid gradient explosion/death in e.g. recurrent nets, while using fewer parameters. And it's not the sort of thing which one easily stumbles across by trying random ideas for nets without some reason to focus on gradient explosion/death (or related instability problems) in particular. The work implies a model of key questions/barriers; it isn't just shooting in the dark.

So this is the sort of guy who can look at a proposal, and say "yeah, that might be valuable" vs "that's not really asking the right question" vs "that would be valuable if it worked, but it will have to somehow deal with ".

Contrast that to the median person in ML these days, who. installed some libraries, loaded some weights, maybe fine-tuned a bit, and generally fiddled with a black box. They don't just lack understanding of what's going on in the black box (nobody knows that), they lack any deep model at all of why things work sometimes but not other times. When trying to evaluate a proposal, they may have some shallow patterns to match against (like "make it bigger"), but mostly they expect any project is roughly-similarly-valuable in expectation modulo its budget; their model of their own field is implicitly "throw lots of random stuff at the wall and see what sticks". Such a person "doesn't know what they're doing", in the way that Yoshua Bengio knows what he's doing.

(Aside: note that I'm not saying that all of Yoshua's models are correct. I'm saying that he has any mental models of depth greater than one, while the median person in ML basically doesn't. Even a wrong general model allows one to try things systematically, update models as one goes, and think about how updates should generalize. Someone without a model has a hard time building any generalizable knowledge at all. It's the difference between someone walking around in a dark room bumping into things and roughly remembering the spots they bumped things but repeatedly bumping into the same wall in different spots because they haven't realized there's a wall there, vs someone walking around in a dark room bumping into things, feeling the shapes of the things, and going "hmm feels like a wall going that way, I should strategize to not run into that same wall repeatedly" (even if they are sometimes wrong about where walls are).)

General Model

Model: "impostor syndrome" is actually correct, in most cases. People correctly realize that they basically don't know what they're doing (in the way that e.g. Bengio knows what he's doing). They feel like they're just LARPing their supposed expertise, because they are just LARPing their supposed expertise.

. and under this model it can still be true that the typical person who feels like an impostor is not actually unskilled/clueless compared to the median person in their field. It's just that (on this model) the median person in most fields is really quite clueless, in the relevant sense. Impostor syndrome is arguably better than the most common alternative, whic...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding strategic deception and deceptive alignment, published by Marius Hobbhahn on September 25, 2023 on The AI Alignment Forum.

The following is the main part of a blog post we just published at Apollo Research. Our main goal with the post is to clarify the concept of deceptive alignment and distinguish it from strategic deception. Furthermore, we want these concepts to become accessible to a non-technical audience such that it is easier for e.g. the general public and policymakers to understand what we're worried about. Feedback is appreciated. For additional sections and appendices, see the full post.

We want AI to always be honest and truthful with us, i.e. we want to prevent situations where the AI model is deceptive about its intentions to its designers or users. Scenarios in which AI models are strategically deceptive could be catastrophic for humanity, e.g. because it could allow AIs that don't have our best interest in mind to get into positions of significant power such as by being deployed in high-stakes settings. Thus, we believe it's crucial to have a clear and comprehensible understanding of AI deception.

In this article, we will describe the concepts of strategic deception and deceptive alignment in detail. In future articles, we will discuss why a model might become deceptive in the first place and how we could evaluate that. Note that we deviate slightly from the definition of deceptive alignment as presented in (Hubinger et al., 2019) for reasons explained in Appendix A.

Core concepts

A colloquial definition of deception is broad and vague. It includes individuals who are sometimes lying, cheating, exaggerating their accomplishments, gaslighting others, and more. We think it would be bad if AIs were to show these traits but we think it would not necessarily be catastrophic.

Thus, we want to distinguish this colloquial definition of deception from strategic deception (SD) and deceptive alignment (DA) which are narrower concepts but, in our opinion, much more likely to lead to catastrophic consequences because the model acts in a more goal-directed manner for SD and DA. We also define what we mean by Alignment, as this affects which cases should and shouldn't be considered deceptive alignment.

For our definitions of deceptive alignment, the concept of "goals" in AIs is important but there is disagreement about what it refers to. For example, if an LLM is prompted to solve a problem, does it have the goal to solve that problem? We would argue that the highest-level goals of the model that are stable across contexts are the ones that matter. In current user-facing LLMs like ChatGPT or Claude, the closest approximation to goals may be being helpful, harmless, and honest. In-context goals can be used to study goal-directed behavior in LLMs but are not what we're primarily interested in.

A model may not have one coherent goal, e.g. it could have a number of incoherent contextually activated preferences (see e.g. Turner & Pope, 2022). In that case, we would say that a model is deceptively aligned for a particular contextually activated preference if it robustly shows deceptive behavior for that preference.

Strategic deception

Attempting to systematically cause a false belief in another entity in order to accomplish some outcome.

This definition, sometimes also called instrumental deception, is adapted from Ward et al., 2023 and Park et al., 2023). Any entity that can be said to have beliefs may be the target of strategic deception, including (groups of) AIs or individuals. The strategic component of SD means that the offending entity is acting in a targeted, goal-directed rather than random manner. For example, the more consistent and well-planned the deception is, the more strategic it is. Thus, strategic deception is on a spectrum and not binary -...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How have you become more hard-working?, published by Chi on September 25, 2023 on The Effective Altruism Forum.

I'd be curious to hear stories of people who have successfully become more hard-working, especially if they started out as not particularly hard-working. Types of things I can imagine playing a role or know have played a role for some people:

Switching roles to something that is conducive to hard work, e.g. a fast-paced environment with lots of concrete tasks and fires to put out.

Medication, e.g. ADHD medication

Internal work, e.g. specific types of therapy, meditation, self-help reading, or other types of reflection.

Productivity hacks, e.g. more accountability, putting specific systems in place

Motivational events, arguments, or life periods, e.g. working a normal corporate jobs where long hours are expected

Switching work environment to something that is conducive to hard work, e.g. always working in an office with others who hold you accountable

This curiosity was triggered by realising that I know of very few people that have become substantially harder-working over their late adolescence/adult life. I also noticed that the few people that I know successfully and seemingly permanently increased their mental health/work satisfaction always were hard-working even when they were unhappy (unless they were in the middle of burn-out or similar).

People becoming more hard-working seems really useful but I haven't seen much in terms of evidence that it's feasible or effective methods. If there are books or studies on this topic, those would also be welcome. Thank you!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org