Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Staring into the abyss as a core life skill, published by benkuhn on December 22, 2022 on LessWrong.

Recently I’ve been thinking about how all my favorite people are great at a skill I’ve labeled in my head as “staring into the abyss.”1

Staring into the abyss means thinking reasonably about things that are uncomfortable to contemplate, like arguments against your religious beliefs, or in favor of breaking up with your partner. It’s common to procrastinate on thinking hard about these things because it might require you to acknowledge that you were very wrong about something in the past, and perhaps wasted a bunch of time based on that (e.g. dating the wrong person or praying to the wrong god). However, in most cases you have to either admit this eventually or, if you never admit it, lock yourself into a sub-optimal future life trajectory, so it’s best to be impatient and stare directly into the uncomfortable topic until you’ve figured out what to do.

The first time I learned what really exceptional abyss-staring looks like, it was by watching Drew, the CEO of Wave. Starting a company requires a lot of staring into the abyss, because it involves making lots of serious mistakes (building the wrong thing, hiring the wrong person, etc.); to move quickly, you need to be fast at acknowledging and fixing them. Drew was extremely willing to tackle uncomfortable decisions head-on—“should we not have hired this person?” “Should we pivot away from this business that is pretty good but not great?"—and every time, it was immediately obvious that the decision he made was a big improvement.

Since then, I’ve become fascinated by the role that abyss-staring plays in people’s lives. I noticed that it wasn’t just Drew who is great at this, but many the people whose work I respect the most, or who have had the most impact on how I think. Conversely, I also noticed that for many of the people I know who have struggled to make good high-level life decisions, they were at least partly blocked by having an abyss that they needed to stare into, but flinched away from.

So I’ve come to believe that becoming more willing to stare into the abyss is one of the most important things you can do to become a better thinker and make better decisions about how to spend your life.

To try to recreate the flavor of watching Drew stare into the abyss for seven years, here are some examples.

When he and his cofounder Lincoln first started a company together (well before my time), they built about 10 different social mobile apps in succession, pivoting from each one when it didn’t get traction. The decision to pivot away from a product you’ve invested a lot of effort into designing, building and distributing is painful enough that many people procrastinate on it and keep working on ideas that clearly aren’t working. In fact, Drew and Lincoln had a third cofounder who left partway through this phase in part because doing so many things that failed made him too stressed and anxious. (To be clear, this is a reasonable reaction that I probably also would have had if I’d been working with them at the time.)

Once they pivoted the 11th time and launched Sendwave—building money transfer from the US to Kenya by delivering to the M-Pesa mobile money system—the product grew incredibly quickly and within less than a year, a majority of the total possible users were using Wave. (I joined toward the tail end of that year.) When we tried to expand to other countries, we realized that their mobile money systems weren’t nearly as good as M-Pesa, which meant that the user experience was worse and the potential market was smaller than expected.

The default response would have been to ignore this info and continue trying to expand Sendwave to integrate with progressively worse mobile money systems in other countries, thus impli...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 1.5X match for Canadian donors to GiveDirectly, published by Jendayi on December 22, 2022 on The Effective Altruism Forum.

Hey everyone, I work on the Growth team at GiveDirectly and wanted to share this match just for Canadian donors running through Dec. 31st 2022.

Only $1,300 of our $75,000 (CAD) match pool has been claimed and this is a true match, so any funds left over after 12/31 will expire.

Donations are matched 1.5X up to $1,500 per donor, meaning a $1,500 donation will unlock $2,250 for people in poverty. That’s enough to deliver large, one-time cash transfers to 2 households to spend and invest as they see fit.

This is a great opportunity to maximize the impact of your giving and enable us to deliver more cash to people living in poverty next year. Consider sharing with your networks (or upvoting this post) to help spread the word.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Holden’s alignment plan, published by Alex Flint on December 22, 2022 on The AI Alignment Forum.

This work was supported by the Monastic Academy for the Preservation of Life on Earth. You can support my work directly here.

I will give a short presentation of this work followed by discussion on Saturday (Dec 24) at 9am Pacific / 12pm Eastern. RSVP here.

Outline

This is a response to Holden Karnofsky’s plan for aligning powerful AI systems in the case that such systems are developed soon.

I give a summary of Holden’s plan, and then critique it.

My basic critique is that the tools Holden is proposing are too similar to what has been tried in the construction of human institutions, and I argue that we have failed to create large numbers of human institutions that alleviate rather than exacerbate existential risk.

The Karnofsky Plan

Holden has written about how we might align very powerful AI systems if such systems were to be developed soon. The basic approach Holden proposes is to train AI systems to be honest, legible, and corrigible, and value-aligned using human supervision. Honesty means not engaging in deceit or manipulation of the supervisor. Legibility means creating artifacts that help supervisors to accurately review decisions. Corrigibility means being willing to stop when the human supervisor says no, no matter how strongly the AI believes that it is doing the right thing. Value-alignment means pursuing the goals of the supervisor. To train such AI systems, Holden says we need to pay attention to the following properties of the training environment:

(1) The human supervisor should provide high-quality feedback. This means that if we are training an AI system to be honest then the human supervisor should give a positive reward if and only if the AI system is, in fact, being honest. It can be hard to know whether an AI system is being honest. Therefore we may design further tools to help the human assess whether the AI system is being honest. In order to do this we may train auxiliary AI systems to assess our primary AI system, or we may restrict the complexity of the primary AI systems, or we may set up an ecosystem of differently-trained AI systems in such a way that dishonesty is "weeded out" by ecological dynamics.

(2) The AI learner should correctly generalize what it has been taught. This means that if we are training an AI system to be honest then the AI system should be honest even when presented with situations very unlike those that it encountered during training. To accomplish this we may identify, during training, which situations are most likely to cause undesirable behavior given the current state of the AI, and then subject the AI to those situations and provide negative rewards until the undesirable behavior is corrected. We may train auxiliary AI systems to help with identifying the situations most likely to cause undesirable behavior given the current state of the AI (adversarial training).

(3) The AI learner should not break out of the training container. This means that if we are training an AI system to be honest, it should be possible for the human supervisor to hide certain things outside the training container, such as schemes for testing the AI system in clever ways. If the AI system escapes the training container, hacks the human supervisor’s cell phone, and listens to the human supervisors discussing the structure of the next test, then the AI system may be able to appear honest without actually being honest. More generally, the AI system should not escape the training container during training. To accomplish this we may build auxiliary AI systems that help to find and patch security holes in the software that holds the training container.

(4) The AI learner should be thoroughly tested before deployment. This means that af...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against philanthropic diversification, published by Jack Malde on December 22, 2022 on The Effective Altruism Forum.

(Cross-posted from my substack The Ethical Economist: a blog covering Economics, Ethics and Effective Altruism.)

Don’t give to just one charity, there are so many good charities to give to! Also, what if the charity turns out to be ineffective? Then you just wasted all your money and did no good. Don’t worry, there’s a simple solution. Just give to multiple charities and spread risk!

Thinking along these lines is natural. Whether it’s risk aversion, or just an inherent desire to support multiple charities or causes, most of us diversify our philanthropic giving. If your goal is to do the most good however, you should fight this urge with all you’ve got.

Diversification does make some sense, some of the time. If you’re going to fill a charity’s budget with your giving then any further giving should probably go elsewhere.

But most of us are small donors. Our giving usually won't fill a budget, or hit diminishing returns. It certainly won’t hit diminishing returns at the level of an entire cause area, unless perhaps you’re a billionaire philanthropist, in which case well done you.

When you’re deciding where to give you likely have some idea of what the best option is. Maybe you want to help animals and are quite uncertain about how best to do so, but you lean towards thinking that giving to The Humane League (THL) to support their corporate campaigns is slightly better on the margin than giving to Faunalytics to support their research, even though you think there’s a possibility either option is ineffective. In this case, you should give your full philanthropic budget to THL. Fight that urge to give to both charities to cover your back if you make the wrong choice.

Giving to both charities reduces the risk of you doing no good. But, because you subjectively think that THL is slightly better than Faunalytics, it also reduces the amount of good you will actually do in expectation. If you think THL is the best, then why give to anything else? Giving to both means trading away expected good done to get more certainty that you yourself will have done some good. It’s putting your own satisfaction ahead of the expected good of the world. Don’t be that person.

At this point you might push back and say that I haven’t convincingly shown that there’s anything wrong with being risk averse in this way. That is, risk averse with respect to the amount of good a particular individual does. Fair enough, so let me try something a bit more formal.

A recent academic paper by Hilary Greaves, William MacAskill, Andreas Mogenson and Teruji Thomas explores the tension between “difference-making risk aversion” and benevolence. Consider the below.

Outcome goodnessHeadsTailsDo nothing100Give to Charity A2010Give to Charity B10+x20+x

A fair coin is to be flipped which determines the payoffs if we do nothing, give to Charity A, or give to Charity B. The coin essentially represents our current uncertainty.

We do have a hunch that giving to Charity B is better. Charity B differs from Charity A in that, instead of a ½ probability of getting 20, Charity B involves a ½ probability of getting 20+x, and instead of a ½ probability of getting 10, Charity B involves a ½ probability of getting 10+x. Given this, it’s clearly better to give to Charity B. In technical language we say that giving to Charity B stochastically dominates giving to Charity A.

Now instead of ‘outcome goodness’ let’s consider the ‘difference made’ of giving to either charity, relative to doing nothing (this is just some simple subtraction using the table above).

Difference madeHeadsTailsDo nothing00Give to Charity A1010Give to Charity Bx20+x

A key thing to notice is that an individual with ‘difference-making risk aversion...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sign up for our Talent Directory if you’re interested in getting a high-impact job, published by High Impact Professionals on December 22, 2022 on The Effective Altruism Forum.

Summary

High Impact Professionals (HIP) is working with high-impact organizations (HIOs) to help them in their recruiting efforts.

To facilitate this effort as well as the efforts of other EA recruiting organizations and HIO hiring managers, we’re building a global EA talent directory.

If you’re interested in working for a HIO, fill out this talent directory sign-up form.

You control your information's privacy as part of completing the form.

Who Should Fill Out The Form?

Anyone who identifies with EA and is looking to transition to working at a HIO. We are especially interested in working professionals as we think they are less legible to the community, but we are happy for any EA to fill it out. If you already filled out our form for FTX grantees that we recently published you don’t need to fill out this form as it is very similar to the one you already filled out.

What Will Happen After I Fill Out the Form?

After you fill out the form you will be entered into our talent directory, a list of all individuals who submit the form. Then, depending on what you’ve consented to on the form, one to two things will happen.

1. We will actively try to place you at a high-impact organization

We at HIP are working with HIOs from diverse cause areas to help them fill roles they are recruiting for. As part of this effort, we will look through our talent directory to find qualified candidates, which hopefully includes you! So the more information you provide to us, the better the chance we have of matching you to a HIO. To do this, we may:

execute a talent search on behalf of a HIO and, in the case of a potential match, either HIP or the HIO will reach out to you.

pass your information to partners who are also interested in either recruiting directly for their organizations or who run a recruiting meta organization as we do. We think this will significantly increase the chances of a match. We will use discretion with whom we share your information; we will also tell all partners not to share your information with anyone further and that they are to use it for the sole purpose of recruiting.

2. We may publish information about you on our website

If you so consent, we may include you in a public talent directory listing on our website so that all organizations looking for talent can find your profile more easily. This offers the least data privacy, but the most publicity for your profile.

You can revoke your consent for either of these options at any time and we will remove your information accordingly. You can also reach out to us to have your information updated.

If this sounds good to you, please sign up to our talent directory.

If you have any comments or suggestions, please feel free to post them below or email them to us.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Keep EA high-trust, published by Michael PJ on December 22, 2022 on The Effective Altruism Forum.

(Status: not draft amnesty, but posting in that spirit, since it's not as good as I'd want it to be but otherwise I probably won't ever post it)

In my experience, EA so far has been a high-trust community. That is, people generally trust other people to behave well and in accordance with the values of the community.

Being high-trust is great! It means that you can spend more time getting on with stuff and less time carefully checking each other for bad behaviour. It's also just nicer: It feels good and motivating to be trusted, and it is reassuring to support people you trust to do work.

I feel like a lot of posts I've seen recently have been arguing for the community to move to a low-trust regime, particularly with respect to EA organizations. That includes calls for:

More transparency ("we need to rigorously scrutinise even your small actions in case you're trying to sneak bad behaviour past us")

More elaborate governance ("there is a risk of governance capture and we need to seriously guard against it", "we don't trust the people currently doing governance")

Sometimes you have to move to low-trust regimes. It's common that organizations tend to move from high-trust to low-trust as they grow, due to the larger number of actors involved who can't all be assumed to be trustworthy. But I do not think that the EA community actually has the problems that require low-trust, and I think it would be very costly.

Specifically, I want to argue:

Low-trust regimes are expensive, both in terms of resources and morale

The people working in current EA orgs are in fact very trustworthy

The EA community should remain high-trust (with checking)

Low-trust is costly

Low-trust regimes impose costs in at least three ways:

Costlier cooperation

Costlier delegation

General efficiency taxes

The post Bad Omens in current EA Governance argues that due to the possibility of conflicts of interest we should break up the organisations which currently share ops support through EVF. This is a clear example of 1: if we can't trust people then we can't just share our resources, we have to keep everyone at arm's length. You can read in the comments various people explaining why this would be quite expensive.

Similarly, you can't just delegate power to people in a low-trust regime. What if they abuse it? Better to require explicit approval up the chain before they do anything serious like spend some money. But if you can't spend money you often can't do things, and activity ends up being blocked on approval, politics, and perception.

When you actually try to get anything done, low-trust regimes typically require lots of paper trails and approvals. Anyone who's worked in a larger organization can testify to how demoralizing and slow this can be. Since any decision can be questioned after the fact, there is no limit to how much "transparency" can be demanded, and how many pointless forms, proposals, reports, or forum posts can end up being produced. I think it is very easy to underestimate how destructive this can be to productivity.

Finally, it is plain demoralizing to be in a low-trust regime. High-trust says "Yes, we are on the same team, go and attack the problem with my blessing!". Low-trust says "I guess I have to work with you but I'm expecting you to try and steal from me as soon as you have the opportunity, so I'm keeping an eye on you". Where would you rather work?

Current people in EA organisations are trustworthy

(Disclaimer: I know quite a lot of people who work in EA organisations, so I'm definitely personally biased towards them.)

The FTX debacle has led to a lot of finger-pointing in recent months. A particular pattern has been posts listing large numbers of questions about the behaviour of p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applied Linear Algebra Lecture Series, published by johnswentworth on December 22, 2022 on LessWrong.

Over the past couple months, I gave weekly lectures on applied linear algebra. The lectures cover a grab-bag of topics which I've needed to know for my own work, but which typically either aren't covered in courses or are covered only briefly in advanced courses which use them (like e.g. quantum). The series is now complete, and recordings of all the lectures are available here.

Be warned: all of the lectures were given with zero review and minimal prep. There are errors. There are poor explanations and too few examples. There are places where I only vaguely gesture at an idea and then say to google it if and when you need it. The flip side is that you will see only things I know off the top of my head - and therefore things which I've found useful enough often enough to remember.

Outline of Topics

Lecture 1

Prototypical use cases of linear algebra

First-order approximation of systems of equations for solving or stability analysis

Second-order approximation of a scalar function in many dimensions for optimization or characterizing of peak/bowl shape

First-order approximation of a dynamical system near a steady state

Principal components of a covariance matrix

Lecture 2

Working with efficient representations of large matrices

Tricks for jacobian and hessian matrices

Prototypical API for implicit matrix representations: scipy's LinearOperator

Lecture 3

Suppose we look at a matrix (e.g. using pyplot.matshow()). What patterns are we most likely to see, and what can we do with them?

Recognizing sparse & low-rank structure

Interpreting sparse & low-rank structure

Leveraging sparse & low-rank structure

Lecture 4

Matrix calculus, with a focus on stability of eigendecomposition

Basics: tensor notation

Differentiating eigendecomposition

Instability of eigenvectors of (approximately) repeated eigenvalues

Lecture 5

Leveraging symmetry

Suppose my system is invariant under some permutation (e.g. a PDE with wraparound boundary, or exchangeable variables in a covariance matrix). How can I leverage that to more efficiently find an eigendecomposition (and invert the matrix etc)?

What Fourier transforms have to do with symmetry, and how to compute them quickly

How to represent rotations/orthogonal matrices

Lecture 6

Wedge products: those "dx dy dz" things in integrals

How to do coordinate transformations with things like "dx dy", even when embedded in a higher-dimensional space

Map between function operations/properties and matrix operations/properties

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 13: RLHF bad, conditioning good., published by Charlie Steiner on December 22, 2022 on The AI Alignment Forum.

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.

Hyperbolic title, sorry. But seriously, conditioning is better than RLHF for current language models. For agents navigating the real world, both have issues and it's not clear-cut where progress will come from.

By "conditioning", I mean the decision transformer trick to do conditional inference: get human ratings of sequences of tokens, and then make a dataset where you append the ratings to the front of the sequences. A model trained on this dataset for next-token prediction will have to learn the distribution of text conditional on the rating - so if you prompt it with a high rating and then the start of an answer, it will try to continue the answer in a way humans would rate highly.

This can be very similar to RLHF - especially if you augment the training data by building a model of human ratings, and train a model to do conditional inference by finetuning a model trained normally. But in the right perspective, the resulting AIs are trying to do quite different things.

RLHF is sorta training the AI to be an agent. Not an agent that navigates the real world, but an agent that navigates the state-space of text. It learns to prefer certain trajectories of the text, and takes actions (outputs words) to steer the text onto favored trajectories. Conditioning, on the other hand, is trying to faithfully learn the distribution of possible human responses - it's getting trained to be a simulator that can predict many different sorts of agents.

The difference is stark in their reactions to variance. RLHF wants to eliminate variance that might make a material difference in the trajectory (when the KL penalty is small relative to the Bayesian-updating KL penalty), while conditioning on rating still tries to produce something that looks like the training distribution.

This makes conditioning way better whenever you care about the diversity of options produced by a language model - e.g. if you're trying to get the AI to generate something specific yet hard to specify, and you want to be able to sift through several continuations. Or if you're building a product that works like souped-up autocorrect, and want to automatically get a diversity of good suggestions.

Another benefit is quantilization. RLHF is trying to get the highest score available, even if it means exploiting human biases. If instead you condition on a score that's high but still regularly gotten by humans, it's like you're sampling policies that get this high-but-not-too-high score, which are less exploitative of human raters than the absolute maximum-score policy.

This isn't a free lunch. Fine-tuning for conditional inference has less of an impact on what sort of problem the AI is solving than RLHF does, but it makes that problem way harder. Unsurprisingly, performance tends to be worse on harder problems. Still, research on decision transformers is full of results that are somewhat competitive with other methods.

It also still exploits the human raters some amount, increasing with the extremity of the score. Sam Marks has talked about a scheme using online decision transformers to improve performance without needing to make the score extreme relative to the distribution seen so far, which is definitely worth a read, but this seems like a case of optimality is the tiger. Whether found by RLHF or conditioning, the problem is with the policies that get the highest scores.

Looking out to the future, I'm uncertain about how useful conditioning will really be. For an AI that chooses policies to affe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Link-post for Caroline Ellison’s Guilty Plea, published by Lauren Maria on December 22, 2022 on The Effective Altruism Forum.

This is a link post for Caroline Ellison’s guilty plea. In addition, the latest from the NY Times is that both her and Gary Wang are co-operating with the federal investigation against Sam Bankman-Fried. Posting here so everyone remains up to date with the progress of this case.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Not Getting Hacked, published by jefftk on December 21, 2022 on LessWrong.

A lot of people get hacked or phished, and being completely secure is somewhere between difficult and impossible, especially if someone trying hard to hack you in particular. But there are several simple things you can do that decrease your risk a lot:

Use a password manager that fills in fields automatically.

Configure 2FA with security keys.

Read "sign in with provider" (oAuth) screens carefully.

Encrypt your laptop.

Be cautious with inbound communication.

Why these things? You can make computers do stuff random other people normally can't: transfer your money, post to social media as you, read your private documents, control your employer's systems, etc. If an attacker can trick the right systems into thinking the are you, they can take your money, exploit your secrets, or use you to get to others.

A strategy of putting in a lot of mental effort to always make the right security decision in the moment doesn't work well for humans: we make mistakes, especially when we're doing routine things or are distracted, tired, hungry, etc. Instead:

Use tools so you can't mess up even if you're not paying

attention.

Understand the remaining cases where you do need to take

special care.

Password managers, laptop encryption, and security keys are examples of the former, while being careful with 2FA codes, oAuth screens, and inbound communication are examples of the latter.

In the most common attacks someone gets a copy of what the computer checks to make sure you are you (your "credentials"). For example, if your email is protected only with a password then if they can get your password they can log into your email.

The most common way for someone to learn your password is using the same one on multiple sites. Say you use "hunter2" everywhere and you create an account on "Joe's Shoes and Wallpaper" to buy sneakers. They later get hacked, their passwords get cracked, and now the attacker knows you used "hunter2". They try this on your email account, learning your address from Joe's records, and it works. This is an especially worrying scenario because you probably have a lot of other accounts that will send password resets to your email, and now the attacker can control those as well.

Remembering passwords for hundreds of sites isn't practical, so you want something that lets you generate a random password for each site and then stores it for you: a "password manager". But this isn't enough on its own, because the second most common risk to passwords is fake login pages. If you click on a link and get:

You want to enter your username and password if this is

accounts.google.com but not if it's

accounts.google.com.evil. Closely examining the URL to make sure it's the right one helps, but humans make mistakes. Instead, you want a password manager that, in addition to remembering your passwords, only provides them to login pages on the correct domain.

I use Google's password manager which is integrated into Chrome and Android; if you're fully in the Apple ecosystem you might use Apple's

Keychain. Firefox also has one. Stand-alone password managers include Bitwarden and 1Password; if you use one of these make sure you set up browser integration so you're not pasting passwords, protecting you against fake login pages. Use separate password managers (or separate "vaults" within a single manager) for work and personal accounts so that when you eventually leave you can separate them cleanly.

Even with an autofilling password manager there will be occasional times when it doesn't work and you need to copy-paste: be very careful that the URL is correct when doing that, because this is exactly the problem you would run into if someone was trying to get your password.

Another layer of protection here is ensuring that ...