Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Giving People Money Doesn’t Help, published by Zvi on July 7, 2022 on LessWrong.

I’ve been thinking a bunch about this study (direct link) of CBT-style techniques used to try and get criminals onto a better life path in Liberia that showed what looks like spectacular results, curious if anyone wants to share thoughts. If it would replicate seems like it crosses the EA-worthiness threshold.

I’ve been thinking more about the topic of this post, which is the cash transfer study in Chicago. That worked out less well. It seems that small cash transfers to those in need here in America made them feel worse and maybe even actually makes them worse off somehow? (paper)

This prediction seems very reasonable, and I think I would have agreed with it if I’d thought about this in advance. My model would have said that obviously cash is good and cash is helpful, in the short term getting some of it will be helpful. In the long term I’d expect some positive impact from the wealth effect. And in some cases, that money will have a particular jam that it solves or opportunity it enables (e.g. fix your car, security deposit on a new apartment, maybe even give opportunity to start a small business). In those cases it would have a permanent impact, so I would predict some overall positive impact. But in my experience there are enough implicit taxes and poverty traps for such people, and such a cultural lack of the idea of hanging onto money, that it is tough to hang onto funds, so mostly they’d be back where they started.

It looks like within about three weeks the money was mostly gone for both groups. That is not what would happen for an economically rational agent, unless they had a lot of opportunities for very high return on capital or they were paying down debt.

Elizabeth had a similar reaction to mine.

There’s something very interesting in this graph. The three groups have completely converged by the end of the 180 day period, but the average bank balance is now considerably higher. They all started out with ~$200 and now they have ~$400. That feels like a meaningfully different average amount.

This happened during Covid, so I am guessing that this is due to the Covid checks or other similar factors. This certainly complicates the experiment.

Those are pretty different things to happen. If you spend the money on bills, that’s paying down debt. It makes sense that this should happen right away. Spending on food and drink in a way you wouldn’t have otherwise is true consumption, the opposite of that.

Note of course this is ‘what they intended to do’ with the money, not what actually happened. Gambling money tends to end up bigger than expected and all that.

I’d ask less ‘what do poor people spend money on?’ and more ‘what do poor people spend more money on when they get a windfall?’

This tells a very clear story. The control group is spending a lot on groceries, but the treatment groups are spending less. I don’t think they are splurging all that much here, more like they are spending money there anyway and money is fungible. This makes sense, this is smart.

Where they are spending it is on bills, utilities (more bills), and especially on housing. Also small on general debt, education and clothes.

Notice that non-durable household goods spending is stronger in the control group, and durable household goods spending is stronger in the treatment group.

Note that it’s not easy to spend only $500 or even $2000 on housing, in a way that you weren’t already spending it, other than paying debt. So my guess is the bulk of this is ‘was behind on rent or mortgage’ and now they aren’t, but the size of this shift says that perhaps some of them did use this to get a new place.

So I interpret this as these people’s balance sheets are shot and they used the money to shore up their balance shee...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human values & biases are inaccessible to the genome, published by Alex Turner on July 7, 2022 on The AI Alignment Forum.

Related to Steve Byrnes’ Social instincts are tricky because of the “symbol grounding problem.” I wouldn’t have had this insight without several great discussions with Quintin Pope.

TL;DR: It seems hard to scan a trained neural network and locate the AI’s learned “tree” abstraction. For very similar reasons, it seems intractable for the genome to scan a human brain and back out the “death” abstraction, which probably will not form at a predictable neural address. Therefore, I infer that the genome can’t directly make us afraid of death by e.g. specifying circuitry which detects when we think about death and then makes us afraid. In turn, this implies that there are a lot of values and biases which the genome cannot hardcode.

In order to understand the human alignment situation confronted by the human genome, consider the AI alignment situation confronted by human civilization. For example, we may want to train a smart AI which learns a sophisticated world model, and then motivate that AI according to its learned world model. Suppose we want to build an AI which intrinsically values trees. Perhaps we can just provide a utility function that queries the learned world model and counts how many trees the AI believes there are.

Suppose that the AI will learn a reasonably human-like concept for “tree.” However, before training has begun, the learned world model is inaccessible to us. Perhaps the learned world model will be buried deep within a recurrent policy network, and buried within the world model is the “trees” concept. But we have no idea what learned circuits will encode that concept, or how the information will be encoded. We probably can’t, in advance of training the AI, write an algorithm which will examine the policy network’s hidden state and reliably back out how many trees the AI thinks there are. The AI’s learned concept for “tree” is inaccessible information from our perspective.

Likewise, the human world model is inaccessible to the human genome, because the world model is probably in the cortex and the cortex is probably randomly initialized. Learned human concepts are therefore inaccessible to the genome, in the same way that the “tree” concept is a priori inaccessible to us. Even the broad area where language processing occurs varies from person to person, to say nothing of the encodings and addresses of particular learned concepts like “death.”

I’m going to say things like “the genome cannot specify circuitry which detects when a person is thinking about death.” This means that the genome cannot hardcode circuitry which e.g. fires when the person is thinking about death, and does not fire when the person is not thinking about death. The genome does help indirectly specify the whole adult brain and all its concepts, just like we indirectly specify the trained neural network via the training algorithm and the dataset. That doesn’t mean we can tell when the AI thinks about trees, and it doesn’t mean that the genome can “tell” when the human thinks about death.

When I’d previously thought about human biases (like the sunk cost fallacy) or values (like caring about other people), I had implicitly imagined that genetic influences could directly affect them (e.g. by detecting when I think about helping my friends, and then producing reward). However, given the inaccessibility obstacle, I infer that this can’t be the explanation. I infer that the genome cannot directly specify circuitry which:

Detects when you’re thinking about seeking power,

Detects when you’re thinking about cheating on your partner,

Detects whether you perceive a sunk cost,

Detects whether you think someone is scamming you and, if so, makes you want to punish them,

Dete...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Person-affecting views can often be Dutch booked, published by Rohin Shah on July 7, 2022 on The Effective Altruism Forum.

This is a short reference post for an argument I wish was better known.

A common intuition people have is that our goal is "Making People Happy, not Making Happy People". That is:

Making people happy: if some person Alice will definitely exist, then it is good to improve her welfare

Not making happy people: it is neutral to go from "Alice won't exist" to "Alice will exist". Intuitively, if Alice doesn't exist, she can't care that she doesn't live a happy life, and so no harm was done.

This position is vulnerable to a Dutch book, that is, there is a set of trades that it would make that would achieve nothing and lose money with certainty. Consider the following worlds:

World 1: Alice won't exist in the future.

World 2: Alice will exist in the future, and will be slightly happy.

World 3: Alice will exist in the future, and will be very happy.

(The worlds are the same in every other aspect. It's a thought experiment.)

Then this view would be happy to make the following trades:

Receive $0.01 to move from World 1 to World 2 ("Not making happy people")

Pay $1.00 to move from World 2 to World 3 ("Making people happy")

Receive $0.01 to move from World 3 to World 1 ("Not making happy people")

The net result is to lose $0.98 to move from World 1 to World 1.

FAQ

Q. Why should I care if my preferences lead to Dutch booking?

This is a longstanding debate that I'm not going to get into here. I'd recommend Holden's series on this general topic, starting with Future-proof ethics.

Q. In the real world we'd never have such clean options to choose from. Why does this matter?

See previous answer.

Q. In step 2, Alice was definitely going to exist, which is why we paid $1. But then in step 3 Alice was no longer definitely going to exist. If we knew step 3 was going to happen, then we wouldn't think Alice was definitely going to exist, and so we wouldn't pay $1.

If your person-affecting view requires people to definitely exist, taking into account all decision-making, then it is almost certainly going to include only currently existing people. This does avoid the Dutch book but has problems of its own, most notably time inconsistency. For example, perhaps right before a baby is born, it take actions that as a side effect will harm the baby; right after the baby is born, it immediately undoes those actions to prevent the side effects.

Q. What if we instead have ?

Often these variants are also vulnerable to the same issue. For example, if you have a "moderate view" where making happy people is not worthless but is discounted by a factor of (say) 10, the same example works with slightly different numbers:

Let's say that "Alice is very happy" has an undiscounted worth of 2 utilons. Then you would be happy to (1) move from World 1 to World 2 for free, (2) pay 1 utilon to move from World 2 to World 3, and (3) receive 0.5 utilons to move from World 3 to World 1.

More generally, Arrhenius proves an impossibility result that applies to all possible population ethics (not just person-affecting views), so (if you want consistency) you need to bite at least one of those bullets.

Further resources

On the Overwhelming Importance of Shaping the Far Future (Nick Beckstead's thesis)

An Impossibility Theorem for Welfarist Axiologies (Arrhenius paradox, summarized in Section 2 of Impossibility and Uncertainty Theorems in AI Value Alignment)

For this post I'll assume that Alice's life is net positive, since "asymmetric" views say that if Alice would have a net negative life, then it would be actively bad (rather than neutral) to move Alice from "won't exist" to "will exist".

By giving it $0.01, I'm making it so that it strictly prefers to take the trade (rather than being indifferent to t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tarnished Guy who Puts a Num on it, published by Jacob Falkovich on July 6, 2022 on LessWrong.

Why Elden Ring is great, and how not to die binomially.Cross-posted from Putanumonit

One reason I haven’t written as much in the last two months is that I’ve been playing a lot of Elden Ring. It’s a video game in which you explore a vast and beautiful open world full of monsters who repeatedly kill you in gruesome ways. It’s a punishing and often frustrating experience that demands skill and discipline to persevere. You wouldn’t think that this is a formula for mass entertainment, but Elden Ring is massively popular. And with good reason.

I’ve written about some of my favorite video games before and what drives the player in each one, whether it’s the addictive minute-by-minute gameplay or a fascinating mystery to uncover as the narrative unforlds. Elden Ring has a background story (co-written by George R. R. Martin) to uncover, but it’s not central to the experience. The gameplay is pretty standard for action-RPGs: you control a character in third-person view, with movement, dodging, and swinging of comically-oversized weapons. Watching a random minute of someone playing Elden Ring doesn’t appear impressive.

Instead, the main goal of playing Elden Ring is to get really good at Elden Ring, because being really good at a very difficult game is incredibly fun. Like other very difficult games, Elden Ring has a progression system with your character upgrading their attributes and gear, but this used just as much to distract the player from the hard work of “gettin gud”. Elden Ring is a rare game where I’m having more fun on my second playthrough — even though my character is weak and has basic gear, the skill I’ve built up lets me breeze through parts that took me long hours to slog through my first time around.

My friend called these games a “stoic self-control fantasy“. Getting good mostly isn’t about committing some complex sequence of finger movements to muscle memory like you would to play a piano piece. Each challenging boss fights requires reacting correctly to one of several possible moves by your adversary, often the opposite of your instinctive reaction the first time you face them. Success requires the self-control to overcome your initial instinct, and to let your skill overcome your fear even as you have no idea from second to second what the next move and counter-move are going to be.

The other thing that success requires is leveling up your vigor so that you don’t get one-shot.

Let’s look in a bit more detail at the game mechanics and the math behind them. Your character in Elden Ring has eight attributes that can be increased with experience. Vigor, mind, and endurance affect the available pools of health, focus (a resource used for special moves), and stamina (a resource for most offensive and defensive actions). The other five attributes affect your ability to use various weapons and skills effectively, like strength for swinging heavy weapons or intelligence for launching powerful spells from long range.

You can’t be good at everything at once, so you must trade off some attributes for others. People plan out “builds”: templates of attribute allocation to pair with particular gear and a desired playing style. You can sacrifice intelligence for strength and build a character that needs to close the distance on an enemy fast, or have a high-faith low-endurance build that needs to constantly evade and disengage to recharge.

But there’s one thing all good builds have in common: they all have high vigor. Because when you lose your health, you die.

Here’s an example of the enemy you face in Elden Ring — a pale asshole wearing the skin of dead gods named, appropriately, the Godskin Asshole Apostle:

GA has nine different attack moves and combinations, each requiring ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Race Along Rashomon Ridge, published by Stephen Fowler on July 7, 2022 on The AI Alignment Forum.

Produced As Part Of The SERI ML Allignment Theory Scholars Program 2022 Research Sprint Under John Wentworth

Two Deep Neural Networks with wildly different parameters can produce equally good results. Not only can a tweak to parameters leave performance unchanged, but in many cases, two neural networks with completely different weights and biases produce identical outputs for any input.

The motivating question:Given two optimal models in a neural network's weight space, is it possible to find a path between them comprised entirely of other optimal models?

In other words, can we find a continuous path of tweaks from the first optimal model to the second without reducing performance at any point in the process?

Ultimately, we hope that the study of equivalently optimal models would lead to advances in interpretability: for example, by producing models that are simultaneously optimal and interpretable.

Introduction

Your friend has invited you to go hiking in the Swiss Alps. Despite extensive planning, you completely failed to clarify which mountain you were supposed to climb. So now you're both currently standing on different mountaintops Communicating via yodels, your friend requests that you hike over to the mountain he's already on; after all, he was the one who organised the trip.

How should you reach your friend? A straight line path would certainly work, but it's such a hassle. Hiking all the way back down and then up another mountain? Madness.

Looking at your map, you notice there's a long ridge that leaves your mountain at roughly the same height. Does this ridge to reach your friend's mountain? possible to follow this ridge to reach your friend's mountain?

Similarly, we can envision two optimal models in a neural net's weight space that share a "ridge" in the loss landscape. There is a natural intuition that we should be able to "walk" between these two points without leaving the ridge. Does this intuition hold up? And what of two neural networks some distance apart in the loss landscape that have both gotten close to the global minima? Can we find a way to walk from one to the other without leaving the ridge?

To begin our investigation into the structure of these "ridges" we viewed these structures as comprising the Rashomon manifold, the subset of the weight space comprised of models that minimize the loss function. In the 2018 paper, Garipov et al. demonstrated that a simple curve is often sufficient to connect two points within the Rashomon manifold.

In our research, we applied the technique of Garipov et al. to path-connect, within a shared weight space, two models that have been trained to be optimal (at an image recognition task with respect to a given dataset). Then, we collected local data along the connecting path. One such local datum is the Hessian of the loss function with respect to the parameters. Its eigendecomposition tells us which directions to travel locally if we want to leave the Rashomon manifold, as well as which directions to travel locally if we want to stay within it. We present a formal description below.

Theory

Consider a neural net.

Let w=(wi)1≤i≤d∈W denote the weight parameters of the trained model yielded by the neural net.

Let L:Rd→R denote the loss function, a real-valued function on the parameter space that the model has been trained to minimize. Assume the loss function L is "nice and smooth''; for example, a neural net that uses skip architecture tends to produce a nice, smooth loss landscape.

We should expect the local minima of a nice loss function to be divided into connected "ridges." Formally, if the loss function is smooth and moreover a submersion, then every level set L−1(c) is a manifold. In particular, the level set of...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Principles for Alignment/Agency Projects, published by johnswentworth on July 7, 2022 on The AI Alignment Forum.

"John, what do you think of this idea for an alignment research project?"

I get questions like that fairly regularly. How do I go about answering? What principles guide my evaluation? Not all of my intuitions for what makes a project valuable can easily be made legible, but I think the principles in this post capture about 80% of the value.

Tackle the Hamming Problems, Don't Avoid Them

Far and away the most common failure mode among self-identifying alignment researchers is to look for Clever Ways To Avoid Doing Hard Things (or Clever Reasons To Ignore The Hard Things), rather than just Directly Tackling The Hard Things.

The most common pattern along these lines is to propose outsourcing the Hard Parts to some future AI, and "just" try to align that AI without understanding the Hard Parts of alignment ourselves. The next most common pattern is to argue that, since Hard Parts are Hard, we definitely don't have enough time to solve them and should therefore pretend that we're going to solve alignment while ignoring them. Third most common is to go into field building, in hopes of getting someone else to solve the Hard Parts. (Admittedly these are not the most charitable summaries.)

There is value in seeing how dumb ideas fail. Most of that value is figuring out what the Hard Parts of the problem are - the taut constraints which we run into over and over again, which we have no idea how to solve. (If it seems pretty solvable, it's probably not a Hard Part.) Once you can recognize the Hard Parts well enough to try to avoid them, you're already past the point where trying dumb ideas has much value.

On a sufficiently new problem, there is also value in checking dumb ideas just in case the problem happens to be easy. Alignment is already past that point; it's not easy.

You can save yourself several years of time and effort by actively trying to identify the Hard Parts and focus on them, rather than avoid them. Otherwise, you'll end up burning several years on ideas which don't actually leave the field better off. That's one of the big problems with trying to circumvent the Hard Parts: when the circumvention inevitably fails, we are still no closer to solving the Hard Parts. (It has been observed both that alignment researchers mostly seem to not be tackling the Hard Parts, and that alignment research mostly doesn't seem to build on itself; I claim that the latter is a result of the former.)

Mostly, I think the hard parts are things like "understand agency in general better" and "understand what's going on inside the magic black boxes". If your response to such things is "sounds hard, man", then you have successfully identified (some of) the Hard Parts.

Have An Intuitive Story Of What We're Looking For

One project going right now is looking at how modularity in trained systems corresponds to broad peaks in parameter space. Intuitive story for that: we have two "modules", each with lots of stuff going on inside, but only a relatively-low-dimensional interface between them. Because each module has lots of stuff going on inside, but only a low-dimensional interface, there should be many ways to change around the insides of a module while keeping the externally-visible behavior the same. Because such changes don't change behavior, they don't change system performance. So, we expect that modularity implies lots of degrees-of-freedom in parameter space, i.e. broad peaks.

This story is way too abstract to be able to look for immediately in a trained net. How do we operationalize "modules", and find them? How do we operationalize "changes in a module", especially since parameter space may not line up very neatly with functional modules? But that's fine; the story can be p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Networking for Nerds, published by Evie Cottrell on July 6, 2022 on The Effective Altruism Forum.

Sharing because I think this post is so damn good and will likely be valuable for at least one person. Originally written by Ben Reinhardt. Check out his blog here; DM him on Twitter here.

“Networking.” What a word. When I left grad school in 2015, networking was for slick-haired salesmen, former jocks, and social parasites in general. Real heroes built better mousetraps and the world beat a path to their doors. But then I looked at what real successes actually do - they lead a balancing act between connecting and building. Though Edison, Einstein, Musk (and anybody you may have heard of) have done real work, their projects would have been stillborn or failed if they didn’t know the right people: partners, employees, backers, supporters.

Always the slave to the scientific method, I proceeded with a “networking might be useful” as a hypothesis and did many experiments. Some were successful, some less so. In the process I developed a set of tactics that unleashed many opportunities - a stint as an EIR at Susa Ventures, speaking at conferences in Brazil and SXSW, and getting a company off the ground. (I like to think that) these were all built on a door of real value, but tactical network building provided the key.

Here are just a few of the specific tactics I’ve developed. At the end of the day, all of them boil down to one thing - make it easy for the other person - to remember you, to help, and to meet.

Tactic 1: The Projection Model

How do some people seem to wind up with advocates everywhere? They create a little mental model of themselves that makes it easy for everybody they meet.

Everybody has a “projection” - a small representation - of each person they know living in their heads. You know that friend who’s “the badass interface designer who is looking to get into machine learning and consults?” She has an amazing projection and makes it easy for opportunities to find her. A projection can be distinct - “Ben is looking for X, Y, Z and can help with A, B, C.” Or it can be fuzzy - “Ben’s that guy with curly hair I think who I met at that meetup and we talked about a bunch of interesting stuff.”

If it’s distinct, whenever X comes up in conversation, you will instantly come to mind, and you’ll start getting messages like “we were having a discussion about X and would love to bring you on as an advisor.”

In my experience, there’s a direct correlation between how well I create a distinct projection and people’s ability to help. When I was “the AR/VR guy who’s interested in giving talks” it led to a new friend introing me to an organization in Brazil that was doing a course on future tech, cascading to lecturing in Sao Paulo, making new friends, and opening up several collaborations.

The trick is that projections are small and can only hold a few pieces of information (rule of thumb - about three.) In order to maximize the effectiveness of your projections, figure out which things you want to be associated with in people’s heads and be excited about them. Nothing is more memorable than distinct excitement. Similarly, try to figure out the things that drive the whomever you’re talking to and incorporate them into your mental representation of that person. The best way to encourage people to keep you in mind is to keep them in mind. (NOTE: Many people take this to the extreme and straight up ask the question: “how can I help?” This is exactly the sentiment you should have, but it makes your counterpart feel like you weren’t actually listening )

Tell a Story

How do you build a strong projection? A strong story. Stories stick in people’s heads better than anything else. Make sure you have a good one. All you need is a short narrative with at least one compelling character (yo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Future Forum - Apply Now, published by isaakfreeman on July 6, 2022 on The Effective Altruism Forum.

We are excited to announce the Future Forum, an experimental 4-day conference taking place Aug 4-7 in San Francisco. We want to gather promising people from across communities interested in the long-term future of humanity, introduce them to each other, reflect on transformative technologies and their far-reaching consequences (upside and downside risks), and then we hope to generate new promising and thoughtful projects across this cross-community ecosystem. We expect around 250 attendees.

We’re putting special emphasis on mingling EA and Silicon Valley tech. We think that Silicon Valley contains promising talent that does or could work on pressing problems with agency and ambition, but that it could also benefit from EA-inspired thoughtfulness about impact and downside risks of working on transformative technologies. On the flip side, we believe EA could generally benefit from more agency and ambition (judicious ambition). We think agency and ambition are best fostered by curated exposure to high-agency individuals, ideally resulting in not just acute inspiration but also more long-term influences on personal trajectories, such as inspiring friends or mentors.

Other communities which will be represented or somewhat involved include Emergent Ventures, Rationalism, Longevity, Crypto, Space, among others.

Generally, the Future Forum will be an in-person-only event, with no virtual component and no talks being recorded.

The Event

While the vast majority of the event is explicitly focused on 1-1s and small group conversations, we have a few speakers who will hold brief talks (~30min max) about topics they care about and give participants common topics to talk about with others. We’ve gotten strong buy-in from many of the communities we hope to bring together for Future Forum, and our current speaker line-up is:

Holden Karnofsky from Open Philanthropy

Anders Sandberg from FHI

Daniela Amodei from Anthropic

Patrick Collison from Stripe

Ed Boyden from MIT/HHMI

Tamara Winter from Stripe Press

Jason Crawford from Roots of Progress

Grant Sanderson (3Blue1Brown)

Allison Duettmann from Foresight Institute

Sam Altman from OpenAI

Celine Halioua from Loyal

(Please be aware that this line-up is likely to evolve further, with more additions from across communities.)

Future Forum runs from August 4th-7th, which is the weekend after EAG San Francisco (EAG’s applications close on July 14th). In addition to EAG SF, there are some other major events and a general concentration of EAs happening in this 2-week time span in the Bay Area, so it might be generally good to come to the Bay around this time.

What we hope to do

We want to positively influence the trajectory of transformative technologies by:

Identifying promising individuals in adjacent communities and providing resources for them to work on important issues

With a focus on the tenet that a disproportionate part of positive progress is driven by few outsized-impact individuals, and thus focussing on potential outsized-impact individuals at the main event and their following career trajectories - e.g. here, or as a principle not dissimilar to hit-based philanthropy.

An example of a community we want to be more in touch with: Many Emergent Ventures grantees seem very promising to do impactful work but might have not interacted with the conventional EA ecosystem by both not having been exposed to the ideas and people much.

A specific example of what it might look like to find a promising individual and accelerate them towards working on a thoughtful, impactful project - ideally one with very asymmetric robust high upside and low downside potential (Asymmetry): the search for superbunker project leads. After identifying ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predicting Parental Emotional Changes?, published by jefftk on July 6, 2022 on LessWrong.

Before I became a parent I cared about children in general, intellectually, in the same way that I care about people in general: a general sense that I want good things to happen to them. Over the first few days of Lily's life, however, I noticed I cared for her deeply and emotionally, where her wellbeing was incredibly important to me, and she was far more cuddly and interesting than I expected. Over the next few months, I realized that my disposition toward children in general had also changed, and while I didn't feel as strongly as toward my own, they were now similarly far more fun, interesting, and precious than they had been. I also started having

strong emotional reactions to reading about harm to children. Paul Graham

writes about a similar experience, though I think one a bit more sudden than my own:

Partly, and I won't deny it, this is because of serious chemical changes that happened almost instantly when our first child was born. It was like someone flipped a switch. I suddenly felt protective not just toward our child, but toward all children.

In deciding to have kids this is a very difficult factor to consider, because it doesn't happen to everyone. I have a vague impression that it is a bit more common in men than women, perhaps because women understandably have a higher threshold for deciding to have kids, but there are definitely men who expect it to happen and are dismayed when it doesn't and women who are surprised at how much their feelings toward children change.

Getting better at predicting this transformation seems valuable: since becoming a parent is a lifelong decision, the value of information is really high. Do people tend to have similar experiences to their own parents? Is it correlated with any other aspects of personality? Can your friends or other people who know you well predict this? Can it be triggered, if it's going to, by something less permanent than becoming a parent or is the permanence a key component? Is it something that may or may not happen, and when it happens it is a similar magnitude for everyone, or is it something where most people change a bit on becoming a parent, anywhere along a range from

"imperceptibly" to "enormously"? Do we know how prevalent these changes are?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Outer vs inner misalignment: three framings, published by Richard Ngo on July 6, 2022 on The AI Alignment Forum.

A core concept in the field of AI alignment is a distinction between two types of misalignment: outer misalignment and inner misalignment. Roughly speaking, the outer alignment problem is the problem of specifying an reward function which captures human preferences; and the inner alignment problem is the problem of ensuring that a policy trained on that reward function actually tries to act in accordance with human preferences. (In other words, it’s the distinction between aligning the “outer” training signal versus aligning the “inner” policy.) However, the distinction can be difficult to pin down precisely. In this post I’ll give three and a half definitions, which each come progressively closer to capturing my current conception of it. I think Framing 1 is a solid starting point; Framings 1.5 and 2 seem like useful refinements, although less concrete; and Framing 3 is fairly speculative.

For those who don’t already have a solid grasp on the inner-outer misalignment distinction, I recommend only reading Framings 1, 1.5 and 2.

For the purposes of this document, I’ll focus on the reinforcement learning setting, since that’s the case where the distinction is clearest. However, the same concepts could also apply in a supervised or self-supervised context, if we replace references to “policies” and “reward functions” with “models” and “loss functions”. The extent to which the inner alignment problem will appear in non-RL contexts is an important open question in alignment.

Framing 1: types of behavioral misalignment

Consider training an RL policy until it’s getting high reward in its training environment. Suppose we then evaluate the policy in a different test environment, without retraining it; what will happen? Consider four possibilities:

The policy behaves incompetently. This is a capability generalization failure.

The policy behaves in a competent and desirable way. This is aligned behavior.

The policy behaves in a competent yet undesirable way which gets high reward according to the original reward function. This is an outer alignment failure, also known as reward misspecification.

The policy behaves in a competent yet undesirable way which gets low reward according to the original reward function. This is an inner alignment failure, also known as goal misgeneralization. Langosco et al. (2022) provide a more formal definition and some examples of goal misgeneralization.

Note that this categorization requires us to evaluate the original reward function in the new environment. This makes sense when the environments are fairly similar (e.g. “playing Go against an AI” vs “playing Go against a human”) or the reward function is fairly general (e.g. based on human evaluations). However, we should be most concerned about alignment failures in policies that are able to behave competently in test environments very different from their training environment (e.g. generalizing from simulations to the real world, or from one task to another). Under such large shifts, there may be many reasonable ways to generalize the original reward function to the test environment (even for human evaluators), which blurs the distinction between reward misspecification and goal misgeneralization. The next framing aims to provide a definition of alignment which avoids this problem.

Framing 1.5: causes of behavioral misalignment

When a policy misbehaves in a test environment, the thing we intuitively care about is what caused that misbehavior. Since we’re not doing any additional training in the test environment, this doesn’t depend on how we evaluate the original reward function in the test environment. Instead, it depends on the rewards the policy received in the training environmen...